Yes, you need to edit the links on your pages, or edit the script that generates those links.
That's step one.
Step two is to write a rule that internally rewrites --"connects"-- those new extensionless URLs, now being requested as a result of your new on-page links, to the correct filepath-with-extension in your server's filesystem.
The third -and optional- step is to detect direct client requests for old URLs that still have extensions, and redirect them to your new extensionless URLs.
If it isn't clear, your old URLs used to be more-or-less the same thing as your filenames. Now they are different. You define the new extensionless URLs on your Web pages, and add mod_rewrite code to re-establish the association between those URLs and your existing filenames.
The optional third step only speeds up the re-indexing of your new URLs and the elimination of your old URLs from search results listings. It is not required.
You can only 'go extensionless' with a very few types of URLs. Trying to allow too many filetypes will bog down your server due to excessive 'file-exists' checking (each of these checks may result in additional reads of the physical disk, which are slow compared to everything else).
Here's an example for .php and .html only:
# Externally redirect direct client requests for URLs ending in .php to extensionless URLs
RewriteCond %{THE_REQUEST} ^[A-Z]+\ /([^/]+/)*[^.]+\.php([?#][^\ ]*)?\ HTTP/
RewriteRule ^(([^/]+/)*[^.]+)\.php$ http://www.example.com/$1 [R=301,L]
#
# Externally redirect direct client requests for URLs ending in .html to extensionless URLs
RewriteCond %{THE_REQUEST} ^[A-Z]+\ /([^/]+/)*[^.]+\.html([?#][^\ ]*)?\ HTTP/
RewriteRule ^(([^/]+/)*[^.]+)\.html$ http://www.example.com/$1 [R=301,L]
#
# Internally rewrite extensionless URLs to add ".php" if a corresponding php file exists
RewriteCond %{REQUEST_FILENAME}.php -f
RewriteRule ^(([^/]+/)*[^.]+)$ /$1.php [L]
#
# Internally rewrite extensionless URLs to add ".html" if a corresponding html file exists
RewriteCond %{REQUEST_FILENAME}.html -f
RewriteRule ^(([^/]+/)*[^.]+)$ /$1.html [L]
Notice that the order of the third and fourth rules establishes a priority of .php files over .html files; If both types of file exist, the php file will be served, and the html file will be inaccessible via HTTP.
The order of the first two rules does not matter, because they are mutually exclusive. You might want to put the rule for the most-frequently-requested filetype first, for a small efficiency improvement.
Note also that the first two rules can be combined, but only if you never rewrite or redirect .html URLs to .php URLs or filepaths, or vice-versa -- now or later.
Jim