Control the document font with the mainfont
variable. E.g., this will use a standard sans-serif TeX font: mainfont: Latin Modern Sans
. Any system-wide font can be used when going via the XeLaTeX engine, i.e., when pandoc is called with --pdf-engine=xelatex
.