I assume you are asking in hosting terms. As you see them at your registrar.
At the registrar where you domain is held, you can choose to have their DNS servers service your domain, or you can chose to customize your DNS server. Like if you want to use an external DNS service, or if you have your own servers, where you want to configure DNS. It merely means you go into their control system and type in your own DNS servers.
So, customized DNS servers simply means that you have asked your registrar to point the DNS for the domain they are holding somewhere else. Whether you own that service (manage it yourself) or not. On Glue.
The host names/domains are of course found by name, which must be translated to an IP.
If the DNS servers for a domain is under the domain you are asking about, like if the server for example.com is ns1.example.com, then you kind of have a circular request.
To find the IP for example.com, you must know the IP for ns1.example.com, which means you need example.com, .., ... Around and around it goes in circles.
To prevent this, DNS delivers Glue records, which are additional records delivered when you ask to have a domain or host name translated. Your registrar's DNS returns the glue, when asked about the domain. Basically points people off to your customized servers. The additional records in the initial answer contains the translation to IP for the real name servers directly, and "Glue" everything together. It is how it is possible to have a domain server connected under the actual domain it is serving.
For glue, for example a 'dig webmasterworld.com' will return
;; QUESTION SECTION:
;webmasterworld.com. IN A
;; ANSWER SECTION:
webmasterworld.com. 135 IN A 188.8.131.52
;; AUTHORITY SECTION:
webmasterworld.com. 172800 IN NS ns1.liquidweb.com.
webmasterworld.com. 172800 IN NS ns2.liquidweb.com.
;; ADDITIONAL SECTION:
ns1.liquidweb.com. 12533 IN A 184.108.40.206
The Authority section tells the caller which name servers officially answer for the domain.
The "Additional Records" section contains the glue records. They tell the system the actual IP address translation for the domain's name servers, so the caller do not need separate requests to figure that out.
The nameservers are then also where the caller would have to go ask more questions about the domain. Such as "Where are your mail-exchangers?', or 'What are your SPF records?'.