It only takes two wires to connect a phone, but most house wiring contains four wires. The red and green wires are on the two center terminals of a normal phone jack, and the black and yellow wires are on the two outer terminals. This arrangement allows most normal houses to handle two phone lines very easily.
Many (if not most) line cords that you receive when you purchase a phone (to go between the wall jack and the phone) only have the two center wires. To save money, the outer two are omitted.
When you bring a second phone line into the house, this new line uses the yellow and black wires. If you have a two-line phone, then the phone is normally ready to accept the second line automatically on the yellow and black wires. You plug one line cord into the phone and it splits the two lines inside the phone. Make sure that the line cord between the jack and the phone actually contains all four wires!
If you want to connect something like a fax machine on the second phone line, you cannot just plug it into the wall. You normally need to buy a little plug-in converter that takes the outer two terminals (yellow/black) of the phone jack and brings them to the center two terminals (red/green). You plug the converter into the wall and the fax machine into the converter.
Because of the proliferation of fax machines, modems, answering machines and other phone-related devices, many new houses are wired so that they can handle up to six phone lines. All six lines are present at each individual jack, and you decide which two the jack will carry by connecting them to the jack's red/green or yellow/black wires.