pdoc.markdown2
A fast and complete Python implementation of Markdown.
[from http://daringfireball.net/projects/markdown/]
Markdown is a text-to-HTML filter; it translates an easy-to-read / easy-to-write structured text format into HTML. Markdown's text format is most similar to that of plain text email, and supports features such as headers, emphasis, code blocks, blockquotes, and links.
Markdown's syntax is designed not as a generic markup language, but specifically to serve as a front-end to (X)HTML. You can use span-level HTML tags anywhere in a Markdown document, and you can use block level HTML tags (like
andas well).
Module usage:
>>> import markdown2 >>> markdown2.markdown("*boo!*") # or use `html = markdown_path(PATH)` u'<p><em>boo!</em></p>\n' >>> markdowner = Markdown() >>> markdowner.convert("*boo!*") u'<p><em>boo!</em></p>\n' >>> markdowner.convert("**boom!**") u'<p><strong>boom!</strong></p>\n'This implementation of Markdown implements the full "core" syntax plus a number of extras (e.g., code syntax coloring, footnotes) as described on https://github.com/trentm/python-markdown2/wiki/Extras.
1# fmt: off 2# flake8: noqa 3# type: ignore 4# Taken from here: https://github.com/trentm/python-markdown2/blob/bce3f18ed86a19b418c8114a712bb6fee790c4c2/lib/markdown2.py 5 6#!/usr/bin/env python 7# Copyright (c) 2012 Trent Mick. 8# Copyright (c) 2007-2008 ActiveState Corp. 9# License: MIT (http://www.opensource.org/licenses/mit-license.php) 10 11r"""A fast and complete Python implementation of Markdown. 12 13[from http://daringfireball.net/projects/markdown/] 14> Markdown is a text-to-HTML filter; it translates an easy-to-read / 15> easy-to-write structured text format into HTML. Markdown's text 16> format is most similar to that of plain text email, and supports 17> features such as headers, *emphasis*, code blocks, blockquotes, and 18> links. 19> 20> Markdown's syntax is designed not as a generic markup language, but 21> specifically to serve as a front-end to (X)HTML. You can use span-level 22> HTML tags anywhere in a Markdown document, and you can use block level 23> HTML tags (like <div> and <table> as well). 24 25Module usage: 26 27 >>> import markdown2 28 >>> markdown2.markdown("*boo!*") # or use `html = markdown_path(PATH)` 29 u'<p><em>boo!</em></p>\n' 30 31 >>> markdowner = Markdown() 32 >>> markdowner.convert("*boo!*") 33 u'<p><em>boo!</em></p>\n' 34 >>> markdowner.convert("**boom!**") 35 u'<p><strong>boom!</strong></p>\n' 36 37This implementation of Markdown implements the full "core" syntax plus a 38number of extras (e.g., code syntax coloring, footnotes) as described on 39<https://github.com/trentm/python-markdown2/wiki/Extras>. 40""" 41 42cmdln_desc = """A fast and complete Python implementation of Markdown, a 43text-to-HTML conversion tool for web writers. 44 45Supported extra syntax options (see -x|--extras option below and 46see <https://github.com/trentm/python-markdown2/wiki/Extras> for details): 47 48* admonitions: Enable parsing of RST admonitions. 49* break-on-newline: Replace single new line characters with <br> when True 50* code-friendly: Disable _ and __ for em and strong. 51* cuddled-lists: Allow lists to be cuddled to the preceding paragraph. 52* fenced-code-blocks: Allows a code block to not have to be indented 53 by fencing it with '```' on a line before and after. Based on 54 <http://github.github.com/github-flavored-markdown/> with support for 55 syntax highlighting. 56* footnotes: Support footnotes as in use on daringfireball.net and 57 implemented in other Markdown processors (tho not in Markdown.pl v1.0.1). 58* header-ids: Adds "id" attributes to headers. The id value is a slug of 59 the header text. 60* highlightjs-lang: Allows specifying the language which used for syntax 61 highlighting when using fenced-code-blocks and highlightjs. 62* html-classes: Takes a dict mapping html tag names (lowercase) to a 63 string to use for a "class" tag attribute. Currently only supports "img", 64 "table", "thead", "pre", "code", "ul" and "ol" tags. Add an issue if you require 65 this for other tags. 66* link-patterns: Auto-link given regex patterns in text (e.g. bug number 67 references, revision number references). 68* markdown-in-html: Allow the use of `markdown="1"` in a block HTML tag to 69 have markdown processing be done on its contents. Similar to 70 <http://michelf.com/projects/php-markdown/extra/#markdown-attr> but with 71 some limitations. 72* metadata: Extract metadata from a leading '---'-fenced block. 73 See <https://github.com/trentm/python-markdown2/issues/77> for details. 74* nofollow: Add `rel="nofollow"` to add `<a>` tags with an href. See 75 <http://en.wikipedia.org/wiki/Nofollow>. 76* numbering: Support of generic counters. Non standard extension to 77 allow sequential numbering of figures, tables, equations, exhibits etc. 78* pyshell: Treats unindented Python interactive shell sessions as <code> 79 blocks. 80* smarty-pants: Replaces ' and " with curly quotation marks or curly 81 apostrophes. Replaces --, ---, ..., and . . . with en dashes, em dashes, 82 and ellipses. 83* spoiler: A special kind of blockquote commonly hidden behind a 84 click on SO. Syntax per <http://meta.stackexchange.com/a/72878>. 85* strike: text inside of double tilde is ~~strikethrough~~ 86* tag-friendly: Requires atx style headers to have a space between the # and 87 the header text. Useful for applications that require twitter style tags to 88 pass through the parser. 89* tables: Tables using the same format as GFM 90 <https://help.github.com/articles/github-flavored-markdown#tables> and 91 PHP-Markdown Extra <https://michelf.ca/projects/php-markdown/extra/#table>. 92* toc: The returned HTML string gets a new "toc_html" attribute which is 93 a Table of Contents for the document. (experimental) 94* use-file-vars: Look for an Emacs-style markdown-extras file variable to turn 95 on Extras. 96* wiki-tables: Google Code Wiki-style tables. See 97 <http://code.google.com/p/support/wiki/WikiSyntax#Tables>. 98* wavedrom: Support for generating Wavedrom digital timing diagrams 99* xml: Passes one-liner processing instructions and namespaced XML tags. 100""" 101 102# Dev Notes: 103# - Python's regex syntax doesn't have '\z', so I'm using '\Z'. I'm 104# not yet sure if there implications with this. Compare 'pydoc sre' 105# and 'perldoc perlre'. 106 107__version_info__ = (2, 4, 9) 108__version__ = '.'.join(map(str, __version_info__)) 109__author__ = "Trent Mick" 110 111import argparse 112import codecs 113import logging 114import re 115import sys 116from collections import defaultdict 117from hashlib import sha256 118from random import randint, random 119 120# ---- globals 121 122DEBUG = False 123log = logging.getLogger("markdown") 124 125DEFAULT_TAB_WIDTH = 4 126 127 128SECRET_SALT = bytes(randint(0, 1000000)) 129# MD5 function was previously used for this; the "md5" prefix was kept for 130# backwards compatibility. 131def _hash_text(s): 132 return 'md5-' + sha256(SECRET_SALT + s.encode("utf-8")).hexdigest()[32:] 133 134# Table of hash values for escaped characters: 135g_escape_table = dict([(ch, _hash_text(ch)) 136 for ch in '\\`*_{}[]()>#+-.!']) 137 138# Ampersand-encoding based entirely on Nat Irons's Amputator MT plugin: 139# http://bumppo.net/projects/amputator/ 140_AMPERSAND_RE = re.compile(r'&(?!#?[xX]?(?:[0-9a-fA-F]+|\w+);)') 141 142 143# ---- exceptions 144class MarkdownError(Exception): 145 pass 146 147 148# ---- public api 149 150def markdown_path(path, encoding="utf-8", 151 html4tags=False, tab_width=DEFAULT_TAB_WIDTH, 152 safe_mode=None, extras=None, link_patterns=None, 153 footnote_title=None, footnote_return_symbol=None, 154 use_file_vars=False): 155 fp = codecs.open(path, 'r', encoding) 156 text = fp.read() 157 fp.close() 158 return Markdown(html4tags=html4tags, tab_width=tab_width, 159 safe_mode=safe_mode, extras=extras, 160 link_patterns=link_patterns, 161 footnote_title=footnote_title, 162 footnote_return_symbol=footnote_return_symbol, 163 use_file_vars=use_file_vars).convert(text) 164 165 166def markdown(text, html4tags=False, tab_width=DEFAULT_TAB_WIDTH, 167 safe_mode=None, extras=None, link_patterns=None, 168 footnote_title=None, footnote_return_symbol=None, 169 use_file_vars=False, cli=False): 170 return Markdown(html4tags=html4tags, tab_width=tab_width, 171 safe_mode=safe_mode, extras=extras, 172 link_patterns=link_patterns, 173 footnote_title=footnote_title, 174 footnote_return_symbol=footnote_return_symbol, 175 use_file_vars=use_file_vars, cli=cli).convert(text) 176 177 178class Markdown(object): 179 # The dict of "extras" to enable in processing -- a mapping of 180 # extra name to argument for the extra. Most extras do not have an 181 # argument, in which case the value is None. 182 # 183 # This can be set via (a) subclassing and (b) the constructor 184 # "extras" argument. 185 extras = None 186 187 urls = None 188 titles = None 189 html_blocks = None 190 html_spans = None 191 html_removed_text = "{(#HTML#)}" # placeholder removed text that does not trigger bold 192 html_removed_text_compat = "[HTML_REMOVED]" # for compat with markdown.py 193 194 _toc = None 195 196 # Used to track when we're inside an ordered or unordered list 197 # (see _ProcessListItems() for details): 198 list_level = 0 199 200 _ws_only_line_re = re.compile(r"^[ \t]+$", re.M) 201 202 def __init__(self, html4tags=False, tab_width=4, safe_mode=None, 203 extras=None, link_patterns=None, 204 footnote_title=None, footnote_return_symbol=None, 205 use_file_vars=False, cli=False): 206 if html4tags: 207 self.empty_element_suffix = ">" 208 else: 209 self.empty_element_suffix = " />" 210 self.tab_width = tab_width 211 self.tab = tab_width * " " 212 213 # For compatibility with earlier markdown2.py and with 214 # markdown.py's safe_mode being a boolean, 215 # safe_mode == True -> "replace" 216 if safe_mode is True: 217 self.safe_mode = "replace" 218 else: 219 self.safe_mode = safe_mode 220 221 # Massaging and building the "extras" info. 222 if self.extras is None: 223 self.extras = {} 224 elif not isinstance(self.extras, dict): 225 self.extras = dict([(e, None) for e in self.extras]) 226 if extras: 227 if not isinstance(extras, dict): 228 extras = dict([(e, None) for e in extras]) 229 self.extras.update(extras) 230 assert isinstance(self.extras, dict) 231 232 if "toc" in self.extras: 233 if "header-ids" not in self.extras: 234 self.extras["header-ids"] = None # "toc" implies "header-ids" 235 236 if self.extras["toc"] is None: 237 self._toc_depth = 6 238 else: 239 self._toc_depth = self.extras["toc"].get("depth", 6) 240 self._instance_extras = self.extras.copy() 241 242 if 'link-patterns' in self.extras: 243 if link_patterns is None: 244 # if you have specified that the link-patterns extra SHOULD 245 # be used (via self.extras) but you haven't provided anything 246 # via the link_patterns argument then an error is raised 247 raise MarkdownError("If the 'link-patterns' extra is used, an argument for 'link_patterns' is required") 248 self.link_patterns = link_patterns 249 self.footnote_title = footnote_title 250 self.footnote_return_symbol = footnote_return_symbol 251 self.use_file_vars = use_file_vars 252 self._outdent_re = re.compile(r'^(\t|[ ]{1,%d})' % tab_width, re.M) 253 self.cli = cli 254 255 self._escape_table = g_escape_table.copy() 256 self._code_table = {} 257 if "smarty-pants" in self.extras: 258 self._escape_table['"'] = _hash_text('"') 259 self._escape_table["'"] = _hash_text("'") 260 261 def reset(self): 262 self.urls = {} 263 self.titles = {} 264 self.html_blocks = {} 265 self.html_spans = {} 266 self.list_level = 0 267 self.extras = self._instance_extras.copy() 268 self._setup_extras() 269 self._toc = None 270 271 def _setup_extras(self): 272 if "footnotes" in self.extras: 273 self.footnotes = {} 274 self.footnote_ids = [] 275 if "header-ids" in self.extras: 276 self._count_from_header_id = defaultdict(int) 277 if "metadata" in self.extras: 278 self.metadata = {} 279 280 # Per <https://developer.mozilla.org/en-US/docs/HTML/Element/a> "rel" 281 # should only be used in <a> tags with an "href" attribute. 282 283 # Opens the linked document in a new window or tab 284 # should only used in <a> tags with an "href" attribute. 285 # same with _a_nofollow 286 _a_nofollow_or_blank_links = re.compile(r""" 287 <(a) 288 ( 289 [^>]* 290 href= # href is required 291 ['"]? # HTML5 attribute values do not have to be quoted 292 [^#'"] # We don't want to match href values that start with # (like footnotes) 293 ) 294 """, 295 re.IGNORECASE | re.VERBOSE 296 ) 297 298 def convert(self, text): 299 """Convert the given text.""" 300 # Main function. The order in which other subs are called here is 301 # essential. Link and image substitutions need to happen before 302 # _EscapeSpecialChars(), so that any *'s or _'s in the <a> 303 # and <img> tags get encoded. 304 305 # Clear the global hashes. If we don't clear these, you get conflicts 306 # from other articles when generating a page which contains more than 307 # one article (e.g. an index page that shows the N most recent 308 # articles): 309 self.reset() 310 311 if not isinstance(text, str): 312 # TODO: perhaps shouldn't presume UTF-8 for string input? 313 text = str(text, 'utf-8') 314 315 if self.use_file_vars: 316 # Look for emacs-style file variable hints. 317 text = self._emacs_oneliner_vars_pat.sub(self._emacs_vars_oneliner_sub, text) 318 emacs_vars = self._get_emacs_vars(text) 319 if "markdown-extras" in emacs_vars: 320 splitter = re.compile("[ ,]+") 321 for e in splitter.split(emacs_vars["markdown-extras"]): 322 if '=' in e: 323 ename, earg = e.split('=', 1) 324 try: 325 earg = int(earg) 326 except ValueError: 327 pass 328 else: 329 ename, earg = e, None 330 self.extras[ename] = earg 331 332 self._setup_extras() 333 334 # Standardize line endings: 335 text = text.replace("\r\n", "\n") 336 text = text.replace("\r", "\n") 337 338 # Make sure $text ends with a couple of newlines: 339 text += "\n\n" 340 341 # Convert all tabs to spaces. 342 text = self._detab(text) 343 344 # Strip any lines consisting only of spaces and tabs. 345 # This makes subsequent regexen easier to write, because we can 346 # match consecutive blank lines with /\n+/ instead of something 347 # contorted like /[ \t]*\n+/ . 348 text = self._ws_only_line_re.sub("", text) 349 350 # strip metadata from head and extract 351 if "metadata" in self.extras: 352 text = self._extract_metadata(text) 353 354 text = self.preprocess(text) 355 356 if 'wavedrom' in self.extras: 357 text = self._do_wavedrom_blocks(text) 358 359 if "fenced-code-blocks" in self.extras and not self.safe_mode: 360 text = self._do_fenced_code_blocks(text) 361 362 if self.safe_mode: 363 text = self._hash_html_spans(text) 364 365 # Turn block-level HTML blocks into hash entries 366 text = self._hash_html_blocks(text, raw=True) 367 368 if "fenced-code-blocks" in self.extras and self.safe_mode: 369 text = self._do_fenced_code_blocks(text) 370 371 if 'admonitions' in self.extras: 372 text = self._do_admonitions(text) 373 374 # Because numbering references aren't links (yet?) then we can do everything associated with counters 375 # before we get started 376 if "numbering" in self.extras: 377 text = self._do_numbering(text) 378 379 # Strip link definitions, store in hashes. 380 if "footnotes" in self.extras: 381 # Must do footnotes first because an unlucky footnote defn 382 # looks like a link defn: 383 # [^4]: this "looks like a link defn" 384 text = self._strip_footnote_definitions(text) 385 text = self._strip_link_definitions(text) 386 387 text = self._run_block_gamut(text) 388 389 if "footnotes" in self.extras: 390 text = self._add_footnotes(text) 391 392 text = self.postprocess(text) 393 394 text = self._unescape_special_chars(text) 395 396 if self.safe_mode: 397 text = self._unhash_html_spans(text) 398 # return the removed text warning to its markdown.py compatible form 399 text = text.replace(self.html_removed_text, self.html_removed_text_compat) 400 401 do_target_blank_links = "target-blank-links" in self.extras 402 do_nofollow_links = "nofollow" in self.extras 403 404 if do_target_blank_links and do_nofollow_links: 405 text = self._a_nofollow_or_blank_links.sub(r'<\1 rel="nofollow noopener" target="_blank"\2', text) 406 elif do_target_blank_links: 407 text = self._a_nofollow_or_blank_links.sub(r'<\1 rel="noopener" target="_blank"\2', text) 408 elif do_nofollow_links: 409 text = self._a_nofollow_or_blank_links.sub(r'<\1 rel="nofollow"\2', text) 410 411 if "toc" in self.extras and self._toc: 412 self._toc_html = calculate_toc_html(self._toc) 413 414 # Prepend toc html to output 415 if self.cli: 416 text = '{}\n{}'.format(self._toc_html, text) 417 418 text += "\n" 419 420 # Attach attrs to output 421 rv = UnicodeWithAttrs(text) 422 423 if "toc" in self.extras and self._toc: 424 rv.toc_html = self._toc_html 425 426 if "metadata" in self.extras: 427 rv.metadata = self.metadata 428 return rv 429 430 def postprocess(self, text): 431 """A hook for subclasses to do some postprocessing of the html, if 432 desired. This is called before unescaping of special chars and 433 unhashing of raw HTML spans. 434 """ 435 return text 436 437 def preprocess(self, text): 438 """A hook for subclasses to do some preprocessing of the Markdown, if 439 desired. This is called after basic formatting of the text, but prior 440 to any extras, safe mode, etc. processing. 441 """ 442 return text 443 444 # Is metadata if the content starts with optional '---'-fenced `key: value` 445 # pairs. E.g. (indented for presentation): 446 # --- 447 # foo: bar 448 # another-var: blah blah 449 # --- 450 # # header 451 # or: 452 # foo: bar 453 # another-var: blah blah 454 # 455 # # header 456 _meta_data_pattern = re.compile(r''' 457 ^{0}( # optional opening fence 458 (?: 459 {1}:(?:\n+[ \t]+.*)+ # indented lists 460 )|(?: 461 (?:{1}:\s+>(?:\n\s+.*)+?) # multiline long descriptions 462 (?=\n{1}:\s*.*\n|\s*\Z) # match up until the start of the next key:value definition or the end of the input text 463 )|(?: 464 {1}:(?! >).*\n? # simple key:value pair, leading spaces allowed 465 ) 466 ){0} # optional closing fence 467 '''.format(r'(?:---[\ \t]*\n)?', r'[\S \t]*\w[\S \t]*\s*'), re.MULTILINE | re.VERBOSE 468 ) 469 470 _key_val_list_pat = re.compile( 471 r"^-(?:[ \t]*([^\n]*)(?:[ \t]*[:-][ \t]*(\S+))?)(?:\n((?:[ \t]+[^\n]+\n?)+))?", 472 re.MULTILINE, 473 ) 474 _key_val_dict_pat = re.compile( 475 r"^([^:\n]+)[ \t]*:[ \t]*([^\n]*)(?:((?:\n[ \t]+[^\n]+)+))?", re.MULTILINE 476 ) # grp0: key, grp1: value, grp2: multiline value 477 _meta_data_fence_pattern = re.compile(r'^---[\ \t]*\n', re.MULTILINE) 478 _meta_data_newline = re.compile("^\n", re.MULTILINE) 479 480 def _extract_metadata(self, text): 481 if text.startswith("---"): 482 fence_splits = re.split(self._meta_data_fence_pattern, text, maxsplit=2) 483 metadata_content = fence_splits[1] 484 match = re.findall(self._meta_data_pattern, metadata_content) 485 if not match: 486 return text 487 tail = fence_splits[2] 488 else: 489 metadata_split = re.split(self._meta_data_newline, text, maxsplit=1) 490 metadata_content = metadata_split[0] 491 match = re.findall(self._meta_data_pattern, metadata_content) 492 if not match: 493 return text 494 tail = metadata_split[1] 495 496 def parse_structured_value(value): 497 vs = value.lstrip() 498 vs = value.replace(v[: len(value) - len(vs)], "\n")[1:] 499 500 # List 501 if vs.startswith("-"): 502 r = [] 503 for match in re.findall(self._key_val_list_pat, vs): 504 if match[0] and not match[1] and not match[2]: 505 r.append(match[0].strip()) 506 elif match[0] == ">" and not match[1] and match[2]: 507 r.append(match[2].strip()) 508 elif match[0] and match[1]: 509 r.append({match[0].strip(): match[1].strip()}) 510 elif not match[0] and not match[1] and match[2]: 511 r.append(parse_structured_value(match[2])) 512 else: 513 # Broken case 514 pass 515 516 return r 517 518 # Dict 519 else: 520 return { 521 match[0].strip(): ( 522 match[1].strip() 523 if match[1] 524 else parse_structured_value(match[2]) 525 ) 526 for match in re.findall(self._key_val_dict_pat, vs) 527 } 528 529 for item in match: 530 531 k, v = item.split(":", 1) 532 533 # Multiline value 534 if v[:3] == " >\n": 535 self.metadata[k.strip()] = _dedent(v[3:]).strip() 536 537 # Empty value 538 elif v == "\n": 539 self.metadata[k.strip()] = "" 540 541 # Structured value 542 elif v[0] == "\n": 543 self.metadata[k.strip()] = parse_structured_value(v) 544 545 # Simple value 546 else: 547 self.metadata[k.strip()] = v.strip() 548 549 return tail 550 551 _emacs_oneliner_vars_pat = re.compile(r"((?:<!--)?\s*-\*-)\s*(?:(\S[^\r\n]*?)([\r\n]\s*)?)?(-\*-\s*(?:-->)?)", re.UNICODE) 552 # This regular expression is intended to match blocks like this: 553 # PREFIX Local Variables: SUFFIX 554 # PREFIX mode: Tcl SUFFIX 555 # PREFIX End: SUFFIX 556 # Some notes: 557 # - "[ \t]" is used instead of "\s" to specifically exclude newlines 558 # - "(\r\n|\n|\r)" is used instead of "$" because the sre engine does 559 # not like anything other than Unix-style line terminators. 560 _emacs_local_vars_pat = re.compile(r"""^ 561 (?P<prefix>(?:[^\r\n|\n|\r])*?) 562 [\ \t]*Local\ Variables:[\ \t]* 563 (?P<suffix>.*?)(?:\r\n|\n|\r) 564 (?P<content>.*?\1End:) 565 """, re.IGNORECASE | re.MULTILINE | re.DOTALL | re.VERBOSE) 566 567 def _emacs_vars_oneliner_sub(self, match): 568 if match.group(1).strip() == '-*-' and match.group(4).strip() == '-*-': 569 lead_ws = re.findall(r'^\s*', match.group(1))[0] 570 tail_ws = re.findall(r'\s*$', match.group(4))[0] 571 return '%s<!-- %s %s %s -->%s' % (lead_ws, '-*-', match.group(2).strip(), '-*-', tail_ws) 572 573 start, end = match.span() 574 return match.string[start: end] 575 576 def _get_emacs_vars(self, text): 577 """Return a dictionary of emacs-style local variables. 578 579 Parsing is done loosely according to this spec (and according to 580 some in-practice deviations from this): 581 http://www.gnu.org/software/emacs/manual/html_node/emacs/Specifying-File-Variables.html#Specifying-File-Variables 582 """ 583 emacs_vars = {} 584 SIZE = pow(2, 13) # 8kB 585 586 # Search near the start for a '-*-'-style one-liner of variables. 587 head = text[:SIZE] 588 if "-*-" in head: 589 match = self._emacs_oneliner_vars_pat.search(head) 590 if match: 591 emacs_vars_str = match.group(2) 592 assert '\n' not in emacs_vars_str 593 emacs_var_strs = [s.strip() for s in emacs_vars_str.split(';') 594 if s.strip()] 595 if len(emacs_var_strs) == 1 and ':' not in emacs_var_strs[0]: 596 # While not in the spec, this form is allowed by emacs: 597 # -*- Tcl -*- 598 # where the implied "variable" is "mode". This form 599 # is only allowed if there are no other variables. 600 emacs_vars["mode"] = emacs_var_strs[0].strip() 601 else: 602 for emacs_var_str in emacs_var_strs: 603 try: 604 variable, value = emacs_var_str.strip().split(':', 1) 605 except ValueError: 606 log.debug("emacs variables error: malformed -*- " 607 "line: %r", emacs_var_str) 608 continue 609 # Lowercase the variable name because Emacs allows "Mode" 610 # or "mode" or "MoDe", etc. 611 emacs_vars[variable.lower()] = value.strip() 612 613 tail = text[-SIZE:] 614 if "Local Variables" in tail: 615 match = self._emacs_local_vars_pat.search(tail) 616 if match: 617 prefix = match.group("prefix") 618 suffix = match.group("suffix") 619 lines = match.group("content").splitlines(0) 620 # print "prefix=%r, suffix=%r, content=%r, lines: %s"\ 621 # % (prefix, suffix, match.group("content"), lines) 622 623 # Validate the Local Variables block: proper prefix and suffix 624 # usage. 625 for i, line in enumerate(lines): 626 if not line.startswith(prefix): 627 log.debug("emacs variables error: line '%s' " 628 "does not use proper prefix '%s'" 629 % (line, prefix)) 630 return {} 631 # Don't validate suffix on last line. Emacs doesn't care, 632 # neither should we. 633 if i != len(lines)-1 and not line.endswith(suffix): 634 log.debug("emacs variables error: line '%s' " 635 "does not use proper suffix '%s'" 636 % (line, suffix)) 637 return {} 638 639 # Parse out one emacs var per line. 640 continued_for = None 641 for line in lines[:-1]: # no var on the last line ("PREFIX End:") 642 if prefix: line = line[len(prefix):] # strip prefix 643 if suffix: line = line[:-len(suffix)] # strip suffix 644 line = line.strip() 645 if continued_for: 646 variable = continued_for 647 if line.endswith('\\'): 648 line = line[:-1].rstrip() 649 else: 650 continued_for = None 651 emacs_vars[variable] += ' ' + line 652 else: 653 try: 654 variable, value = line.split(':', 1) 655 except ValueError: 656 log.debug("local variables error: missing colon " 657 "in local variables entry: '%s'" % line) 658 continue 659 # Do NOT lowercase the variable name, because Emacs only 660 # allows "mode" (and not "Mode", "MoDe", etc.) in this block. 661 value = value.strip() 662 if value.endswith('\\'): 663 value = value[:-1].rstrip() 664 continued_for = variable 665 else: 666 continued_for = None 667 emacs_vars[variable] = value 668 669 # Unquote values. 670 for var, val in list(emacs_vars.items()): 671 if len(val) > 1 and (val.startswith('"') and val.endswith('"') 672 or val.startswith('"') and val.endswith('"')): 673 emacs_vars[var] = val[1:-1] 674 675 return emacs_vars 676 677 def _detab_line(self, line): 678 r"""Recusively convert tabs to spaces in a single line. 679 680 Called from _detab().""" 681 if '\t' not in line: 682 return line 683 chunk1, chunk2 = line.split('\t', 1) 684 chunk1 += (' ' * (self.tab_width - len(chunk1) % self.tab_width)) 685 output = chunk1 + chunk2 686 return self._detab_line(output) 687 688 def _detab(self, text): 689 r"""Iterate text line by line and convert tabs to spaces. 690 691 >>> m = Markdown() 692 >>> m._detab("\tfoo") 693 ' foo' 694 >>> m._detab(" \tfoo") 695 ' foo' 696 >>> m._detab("\t foo") 697 ' foo' 698 >>> m._detab(" foo") 699 ' foo' 700 >>> m._detab(" foo\n\tbar\tblam") 701 ' foo\n bar blam' 702 """ 703 if '\t' not in text: 704 return text 705 output = [] 706 for line in text.splitlines(): 707 output.append(self._detab_line(line)) 708 return '\n'.join(output) 709 710 # I broke out the html5 tags here and add them to _block_tags_a and 711 # _block_tags_b. This way html5 tags are easy to keep track of. 712 _html5tags = '|article|aside|header|hgroup|footer|nav|section|figure|figcaption' 713 714 _block_tags_a = 'p|div|h[1-6]|blockquote|pre|table|dl|ol|ul|script|noscript|form|fieldset|iframe|math|ins|del' 715 _block_tags_a += _html5tags 716 717 _strict_tag_block_re = re.compile(r""" 718 ( # save in \1 719 ^ # start of line (with re.M) 720 <(%s) # start tag = \2 721 \b # word break 722 (.*\n)*? # any number of lines, minimally matching 723 </\2> # the matching end tag 724 [ \t]* # trailing spaces/tabs 725 (?=\n+|\Z) # followed by a newline or end of document 726 ) 727 """ % _block_tags_a, 728 re.X | re.M) 729 730 _block_tags_b = 'p|div|h[1-6]|blockquote|pre|table|dl|ol|ul|script|noscript|form|fieldset|iframe|math' 731 _block_tags_b += _html5tags 732 733 _liberal_tag_block_re = re.compile(r""" 734 ( # save in \1 735 ^ # start of line (with re.M) 736 <(%s) # start tag = \2 737 \b # word break 738 (.*\n)*? # any number of lines, minimally matching 739 .*</\2> # the matching end tag 740 [ \t]* # trailing spaces/tabs 741 (?=\n+|\Z) # followed by a newline or end of document 742 ) 743 """ % _block_tags_b, 744 re.X | re.M) 745 746 _html_markdown_attr_re = re.compile( 747 r'''\s+markdown=("1"|'1')''') 748 def _hash_html_block_sub(self, match, raw=False): 749 if isinstance(match, str): 750 html = match 751 else: 752 html = match.group(1) 753 754 if raw and self.safe_mode: 755 html = self._sanitize_html(html) 756 elif 'markdown-in-html' in self.extras and 'markdown=' in html: 757 first_line = html.split('\n', 1)[0] 758 m = self._html_markdown_attr_re.search(first_line) 759 if m: 760 lines = html.split('\n') 761 middle = '\n'.join(lines[1:-1]) 762 last_line = lines[-1] 763 first_line = first_line[:m.start()] + first_line[m.end():] 764 f_key = _hash_text(first_line) 765 self.html_blocks[f_key] = first_line 766 l_key = _hash_text(last_line) 767 self.html_blocks[l_key] = last_line 768 return ''.join(["\n\n", f_key, 769 "\n\n", middle, "\n\n", 770 l_key, "\n\n"]) 771 key = _hash_text(html) 772 self.html_blocks[key] = html 773 return "\n\n" + key + "\n\n" 774 775 def _hash_html_blocks(self, text, raw=False): 776 """Hashify HTML blocks 777 778 We only want to do this for block-level HTML tags, such as headers, 779 lists, and tables. That's because we still want to wrap <p>s around 780 "paragraphs" that are wrapped in non-block-level tags, such as anchors, 781 phrase emphasis, and spans. The list of tags we're looking for is 782 hard-coded. 783 784 @param raw {boolean} indicates if these are raw HTML blocks in 785 the original source. It makes a difference in "safe" mode. 786 """ 787 if '<' not in text: 788 return text 789 790 # Pass `raw` value into our calls to self._hash_html_block_sub. 791 hash_html_block_sub = _curry(self._hash_html_block_sub, raw=raw) 792 793 # First, look for nested blocks, e.g.: 794 # <div> 795 # <div> 796 # tags for inner block must be indented. 797 # </div> 798 # </div> 799 # 800 # The outermost tags must start at the left margin for this to match, and 801 # the inner nested divs must be indented. 802 # We need to do this before the next, more liberal match, because the next 803 # match will start at the first `<div>` and stop at the first `</div>`. 804 text = self._strict_tag_block_sub(text, self._block_tags_a, hash_html_block_sub) 805 806 # Now match more liberally, simply from `\n<tag>` to `</tag>\n` 807 text = self._liberal_tag_block_re.sub(hash_html_block_sub, text) 808 809 # Special case just for <hr />. It was easier to make a special 810 # case than to make the other regex more complicated. 811 if "<hr" in text: 812 _hr_tag_re = _hr_tag_re_from_tab_width(self.tab_width) 813 text = _hr_tag_re.sub(hash_html_block_sub, text) 814 815 # Special case for standalone HTML comments: 816 if "<!--" in text: 817 start = 0 818 while True: 819 # Delimiters for next comment block. 820 try: 821 start_idx = text.index("<!--", start) 822 except ValueError: 823 break 824 try: 825 end_idx = text.index("-->", start_idx) + 3 826 except ValueError: 827 break 828 829 # Start position for next comment block search. 830 start = end_idx 831 832 # Validate whitespace before comment. 833 if start_idx: 834 # - Up to `tab_width - 1` spaces before start_idx. 835 for i in range(self.tab_width - 1): 836 if text[start_idx - 1] != ' ': 837 break 838 start_idx -= 1 839 if start_idx == 0: 840 break 841 # - Must be preceded by 2 newlines or hit the start of 842 # the document. 843 if start_idx == 0: 844 pass 845 elif start_idx == 1 and text[0] == '\n': 846 start_idx = 0 # to match minute detail of Markdown.pl regex 847 elif text[start_idx-2:start_idx] == '\n\n': 848 pass 849 else: 850 break 851 852 # Validate whitespace after comment. 853 # - Any number of spaces and tabs. 854 while end_idx < len(text): 855 if text[end_idx] not in ' \t': 856 break 857 end_idx += 1 858 # - Must be following by 2 newlines or hit end of text. 859 if text[end_idx:end_idx+2] not in ('', '\n', '\n\n'): 860 continue 861 862 # Escape and hash (must match `_hash_html_block_sub`). 863 html = text[start_idx:end_idx] 864 if raw and self.safe_mode: 865 html = self._sanitize_html(html) 866 key = _hash_text(html) 867 self.html_blocks[key] = html 868 text = text[:start_idx] + "\n\n" + key + "\n\n" + text[end_idx:] 869 870 if "xml" in self.extras: 871 # Treat XML processing instructions and namespaced one-liner 872 # tags as if they were block HTML tags. E.g., if standalone 873 # (i.e. are their own paragraph), the following do not get 874 # wrapped in a <p> tag: 875 # <?foo bar?> 876 # 877 # <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="chapter_1.md"/> 878 _xml_oneliner_re = _xml_oneliner_re_from_tab_width(self.tab_width) 879 text = _xml_oneliner_re.sub(hash_html_block_sub, text) 880 881 return text 882 883 def _strict_tag_block_sub(self, text, html_tags_re, callback): 884 tag_count = 0 885 current_tag = html_tags_re 886 block = '' 887 result = '' 888 889 for chunk in text.splitlines(True): 890 is_markup = re.match(r'^(?:</code>(?=</pre>))?(</?(%s)\b>?)' % current_tag, chunk) 891 block += chunk 892 893 if is_markup: 894 if chunk.startswith('</'): 895 tag_count -= 1 896 else: 897 # if close tag is in same line 898 if '</%s>' % is_markup.group(2) in chunk[is_markup.end():]: 899 # we must ignore these 900 is_markup = None 901 else: 902 tag_count += 1 903 current_tag = is_markup.group(2) 904 905 if tag_count == 0: 906 if is_markup: 907 block = callback(block.rstrip('\n')) # remove trailing newline 908 current_tag = html_tags_re 909 result += block 910 block = '' 911 912 result += block 913 914 return result 915 916 def _strip_link_definitions(self, text): 917 # Strips link definitions from text, stores the URLs and titles in 918 # hash references. 919 less_than_tab = self.tab_width - 1 920 921 # Link defs are in the form: 922 # [id]: url "optional title" 923 _link_def_re = re.compile(r""" 924 ^[ ]{0,%d}\[(.+)\]: # id = \1 925 [ \t]* 926 \n? # maybe *one* newline 927 [ \t]* 928 <?(.+?)>? # url = \2 929 [ \t]* 930 (?: 931 \n? # maybe one newline 932 [ \t]* 933 (?<=\s) # lookbehind for whitespace 934 ['"(] 935 ([^\n]*) # title = \3 936 ['")] 937 [ \t]* 938 )? # title is optional 939 (?:\n+|\Z) 940 """ % less_than_tab, re.X | re.M | re.U) 941 return _link_def_re.sub(self._extract_link_def_sub, text) 942 943 def _extract_link_def_sub(self, match): 944 id, url, title = match.groups() 945 key = id.lower() # Link IDs are case-insensitive 946 self.urls[key] = self._encode_amps_and_angles(url) 947 if title: 948 self.titles[key] = title 949 return "" 950 951 def _do_numbering(self, text): 952 ''' We handle the special extension for generic numbering for 953 tables, figures etc. 954 ''' 955 # First pass to define all the references 956 self.regex_defns = re.compile(r''' 957 \[\#(\w+) # the counter. Open square plus hash plus a word \1 958 ([^@]*) # Some optional characters, that aren't an @. \2 959 @(\w+) # the id. Should this be normed? \3 960 ([^\]]*)\] # The rest of the text up to the terminating ] \4 961 ''', re.VERBOSE) 962 self.regex_subs = re.compile(r"\[@(\w+)\s*\]") # [@ref_id] 963 counters = {} 964 references = {} 965 replacements = [] 966 definition_html = '<figcaption class="{}" id="counter-ref-{}">{}{}{}</figcaption>' 967 reference_html = '<a class="{}" href="#counter-ref-{}">{}</a>' 968 for match in self.regex_defns.finditer(text): 969 # We must have four match groups otherwise this isn't a numbering reference 970 if len(match.groups()) != 4: 971 continue 972 counter = match.group(1) 973 text_before = match.group(2).strip() 974 ref_id = match.group(3) 975 text_after = match.group(4) 976 number = counters.get(counter, 1) 977 references[ref_id] = (number, counter) 978 replacements.append((match.start(0), 979 definition_html.format(counter, 980 ref_id, 981 text_before, 982 number, 983 text_after), 984 match.end(0))) 985 counters[counter] = number + 1 986 for repl in reversed(replacements): 987 text = text[:repl[0]] + repl[1] + text[repl[2]:] 988 989 # Second pass to replace the references with the right 990 # value of the counter 991 # Fwiw, it's vaguely annoying to have to turn the iterator into 992 # a list and then reverse it but I can't think of a better thing to do. 993 for match in reversed(list(self.regex_subs.finditer(text))): 994 number, counter = references.get(match.group(1), (None, None)) 995 if number is not None: 996 repl = reference_html.format(counter, 997 match.group(1), 998 number) 999 else: 1000 repl = reference_html.format(match.group(1), 1001 'countererror', 1002 '?' + match.group(1) + '?') 1003 if "smarty-pants" in self.extras: 1004 repl = repl.replace('"', self._escape_table['"']) 1005 1006 text = text[:match.start()] + repl + text[match.end():] 1007 return text 1008 1009 def _extract_footnote_def_sub(self, match): 1010 id, text = match.groups() 1011 text = _dedent(text, skip_first_line=not text.startswith('\n')).strip() 1012 normed_id = re.sub(r'\W', '-', id) 1013 # Ensure footnote text ends with a couple newlines (for some 1014 # block gamut matches). 1015 self.footnotes[normed_id] = text + "\n\n" 1016 return "" 1017 1018 def _strip_footnote_definitions(self, text): 1019 """A footnote definition looks like this: 1020 1021 [^note-id]: Text of the note. 1022 1023 May include one or more indented paragraphs. 1024 1025 Where, 1026 - The 'note-id' can be pretty much anything, though typically it 1027 is the number of the footnote. 1028 - The first paragraph may start on the next line, like so: 1029 1030 [^note-id]: 1031 Text of the note. 1032 """ 1033 less_than_tab = self.tab_width - 1 1034 footnote_def_re = re.compile(r''' 1035 ^[ ]{0,%d}\[\^(.+)\]: # id = \1 1036 [ \t]* 1037 ( # footnote text = \2 1038 # First line need not start with the spaces. 1039 (?:\s*.*\n+) 1040 (?: 1041 (?:[ ]{%d} | \t) # Subsequent lines must be indented. 1042 .*\n+ 1043 )* 1044 ) 1045 # Lookahead for non-space at line-start, or end of doc. 1046 (?:(?=^[ ]{0,%d}\S)|\Z) 1047 ''' % (less_than_tab, self.tab_width, self.tab_width), 1048 re.X | re.M) 1049 return footnote_def_re.sub(self._extract_footnote_def_sub, text) 1050 1051 _hr_re = re.compile(r'^[ ]{0,3}([-_*])[ ]{0,2}(\1[ ]{0,2}){2,}$', re.M) 1052 1053 def _run_block_gamut(self, text): 1054 # These are all the transformations that form block-level 1055 # tags like paragraphs, headers, and list items. 1056 1057 if 'admonitions' in self.extras: 1058 text = self._do_admonitions(text) 1059 1060 if 'wavedrom' in self.extras: 1061 text = self._do_wavedrom_blocks(text) 1062 1063 if "fenced-code-blocks" in self.extras: 1064 text = self._do_fenced_code_blocks(text) 1065 1066 text = self._do_headers(text) 1067 1068 # Do Horizontal Rules: 1069 # On the number of spaces in horizontal rules: The spec is fuzzy: "If 1070 # you wish, you may use spaces between the hyphens or asterisks." 1071 # Markdown.pl 1.0.1's hr regexes limit the number of spaces between the 1072 # hr chars to one or two. We'll reproduce that limit here. 1073 hr = "\n<hr"+self.empty_element_suffix+"\n" 1074 text = re.sub(self._hr_re, hr, text) 1075 1076 text = self._do_lists(text) 1077 1078 if "pyshell" in self.extras: 1079 text = self._prepare_pyshell_blocks(text) 1080 if "wiki-tables" in self.extras: 1081 text = self._do_wiki_tables(text) 1082 if "tables" in self.extras: 1083 text = self._do_tables(text) 1084 1085 text = self._do_code_blocks(text) 1086 1087 text = self._do_block_quotes(text) 1088 1089 # We already ran _HashHTMLBlocks() before, in Markdown(), but that 1090 # was to escape raw HTML in the original Markdown source. This time, 1091 # we're escaping the markup we've just created, so that we don't wrap 1092 # <p> tags around block-level tags. 1093 text = self._hash_html_blocks(text) 1094 1095 text = self._form_paragraphs(text) 1096 1097 return text 1098 1099 def _pyshell_block_sub(self, match): 1100 if "fenced-code-blocks" in self.extras: 1101 dedented = _dedent(match.group(0)) 1102 return self._do_fenced_code_blocks("```pycon\n" + dedented + "```\n") 1103 lines = match.group(0).splitlines(0) 1104 _dedentlines(lines) 1105 indent = ' ' * self.tab_width 1106 s = ('\n' # separate from possible cuddled paragraph 1107 + indent + ('\n'+indent).join(lines) 1108 + '\n') 1109 return s 1110 1111 def _prepare_pyshell_blocks(self, text): 1112 """Ensure that Python interactive shell sessions are put in 1113 code blocks -- even if not properly indented. 1114 """ 1115 if ">>>" not in text: 1116 return text 1117 1118 less_than_tab = self.tab_width - 1 1119 _pyshell_block_re = re.compile(r""" 1120 ^([ ]{0,%d})>>>[ ].*\n # first line 1121 ^(\1[^\S\n]*\S.*\n)* # any number of subsequent lines with at least one character 1122 (?=^\1?\n|\Z) # ends with a blank line or end of document 1123 """ % less_than_tab, re.M | re.X) 1124 1125 return _pyshell_block_re.sub(self._pyshell_block_sub, text) 1126 1127 def _table_sub(self, match): 1128 trim_space_re = '^[ \t\n]+|[ \t\n]+$' 1129 trim_bar_re = r'^\||\|$' 1130 split_bar_re = r'^\||(?<![\`\\])\|' 1131 escape_bar_re = r'\\\|' 1132 1133 head, underline, body = match.groups() 1134 1135 # Determine aligns for columns. 1136 cols = [re.sub(escape_bar_re, '|', cell.strip()) for cell in re.split(split_bar_re, re.sub(trim_bar_re, "", re.sub(trim_space_re, "", underline)))] 1137 align_from_col_idx = {} 1138 for col_idx, col in enumerate(cols): 1139 if col[0] == ':' and col[-1] == ':': 1140 align_from_col_idx[col_idx] = ' style="text-align:center;"' 1141 elif col[0] == ':': 1142 align_from_col_idx[col_idx] = ' style="text-align:left;"' 1143 elif col[-1] == ':': 1144 align_from_col_idx[col_idx] = ' style="text-align:right;"' 1145 1146 # thead 1147 hlines = ['<table%s>' % self._html_class_str_from_tag('table'), '<thead%s>' % self._html_class_str_from_tag('thead'), '<tr>'] 1148 cols = [re.sub(escape_bar_re, '|', cell.strip()) for cell in re.split(split_bar_re, re.sub(trim_bar_re, "", re.sub(trim_space_re, "", head)))] 1149 for col_idx, col in enumerate(cols): 1150 hlines.append(' <th%s>%s</th>' % ( 1151 align_from_col_idx.get(col_idx, ''), 1152 self._run_span_gamut(col) 1153 )) 1154 hlines.append('</tr>') 1155 hlines.append('</thead>') 1156 1157 # tbody 1158 hlines.append('<tbody>') 1159 for line in body.strip('\n').split('\n'): 1160 hlines.append('<tr>') 1161 cols = [re.sub(escape_bar_re, '|', cell.strip()) for cell in re.split(split_bar_re, re.sub(trim_bar_re, "", re.sub(trim_space_re, "", line)))] 1162 for col_idx, col in enumerate(cols): 1163 hlines.append(' <td%s>%s</td>' % ( 1164 align_from_col_idx.get(col_idx, ''), 1165 self._run_span_gamut(col) 1166 )) 1167 hlines.append('</tr>') 1168 hlines.append('</tbody>') 1169 hlines.append('</table>') 1170 1171 return '\n'.join(hlines) + '\n' 1172 1173 def _do_tables(self, text): 1174 """Copying PHP-Markdown and GFM table syntax. Some regex borrowed from 1175 https://github.com/michelf/php-markdown/blob/lib/Michelf/Markdown.php#L2538 1176 """ 1177 less_than_tab = self.tab_width - 1 1178 table_re = re.compile(r''' 1179 (?:(?<=\n\n)|\A\n?) # leading blank line 1180 1181 ^[ ]{0,%d} # allowed whitespace 1182 (.*[|].*) \n # $1: header row (at least one pipe) 1183 1184 ^[ ]{0,%d} # allowed whitespace 1185 ( # $2: underline row 1186 # underline row with leading bar 1187 (?: \|\ *:?-+:?\ * )+ \|? \s? \n 1188 | 1189 # or, underline row without leading bar 1190 (?: \ *:?-+:?\ *\| )+ (?: \ *:?-+:?\ * )? \s? \n 1191 ) 1192 1193 ( # $3: data rows 1194 (?: 1195 ^[ ]{0,%d}(?!\ ) # ensure line begins with 0 to less_than_tab spaces 1196 .*\|.* \n 1197 )+ 1198 ) 1199 ''' % (less_than_tab, less_than_tab, less_than_tab), re.M | re.X) 1200 return table_re.sub(self._table_sub, text) 1201 1202 def _wiki_table_sub(self, match): 1203 ttext = match.group(0).strip() 1204 # print('wiki table: %r' % match.group(0)) 1205 rows = [] 1206 for line in ttext.splitlines(0): 1207 line = line.strip()[2:-2].strip() 1208 row = [c.strip() for c in re.split(r'(?<!\\)\|\|', line)] 1209 rows.append(row) 1210 # from pprint import pprint 1211 # pprint(rows) 1212 hlines = [] 1213 1214 def add_hline(line, indents=0): 1215 hlines.append((self.tab * indents) + line) 1216 1217 def format_cell(text): 1218 return self._run_span_gamut(re.sub(r"^\s*~", "", cell).strip(" ")) 1219 1220 add_hline('<table%s>' % self._html_class_str_from_tag('table')) 1221 # Check if first cell of first row is a header cell. If so, assume the whole row is a header row. 1222 if rows and rows[0] and re.match(r"^\s*~", rows[0][0]): 1223 add_hline('<thead%s>' % self._html_class_str_from_tag('thead'), 1) 1224 add_hline('<tr>', 2) 1225 for cell in rows[0]: 1226 add_hline("<th>{}</th>".format(format_cell(cell)), 3) 1227 add_hline('</tr>', 2) 1228 add_hline('</thead>', 1) 1229 # Only one header row allowed. 1230 rows = rows[1:] 1231 # If no more rows, don't create a tbody. 1232 if rows: 1233 add_hline('<tbody>', 1) 1234 for row in rows: 1235 add_hline('<tr>', 2) 1236 for cell in row: 1237 add_hline('<td>{}</td>'.format(format_cell(cell)), 3) 1238 add_hline('</tr>', 2) 1239 add_hline('</tbody>', 1) 1240 add_hline('</table>') 1241 return '\n'.join(hlines) + '\n' 1242 1243 def _do_wiki_tables(self, text): 1244 # Optimization. 1245 if "||" not in text: 1246 return text 1247 1248 less_than_tab = self.tab_width - 1 1249 wiki_table_re = re.compile(r''' 1250 (?:(?<=\n\n)|\A\n?) # leading blank line 1251 ^([ ]{0,%d})\|\|.+?\|\|[ ]*\n # first line 1252 (^\1\|\|.+?\|\|\n)* # any number of subsequent lines 1253 ''' % less_than_tab, re.M | re.X) 1254 return wiki_table_re.sub(self._wiki_table_sub, text) 1255 1256 def _run_span_gamut(self, text): 1257 # These are all the transformations that occur *within* block-level 1258 # tags like paragraphs, headers, and list items. 1259 1260 text = self._do_code_spans(text) 1261 1262 text = self._escape_special_chars(text) 1263 1264 # Process anchor and image tags. 1265 if "link-patterns" in self.extras: 1266 text = self._do_link_patterns(text) 1267 1268 text = self._do_links(text) 1269 1270 # Make links out of things like `<http://example.com/>` 1271 # Must come after _do_links(), because you can use < and > 1272 # delimiters in inline links like [this](<url>). 1273 text = self._do_auto_links(text) 1274 1275 text = self._encode_amps_and_angles(text) 1276 1277 if "strike" in self.extras: 1278 text = self._do_strike(text) 1279 1280 if "underline" in self.extras: 1281 text = self._do_underline(text) 1282 1283 text = self._do_italics_and_bold(text) 1284 1285 if "tg-spoiler" in self.extras: 1286 text = self._do_tg_spoiler(text) 1287 1288 if "smarty-pants" in self.extras: 1289 text = self._do_smart_punctuation(text) 1290 1291 # Do hard breaks: 1292 if "break-on-newline" in self.extras: 1293 text = re.sub(r" *\n(?!\<(?:\/?(ul|ol|li))\>)", "<br%s\n" % self.empty_element_suffix, text) 1294 else: 1295 text = re.sub(r" {2,}\n", " <br%s\n" % self.empty_element_suffix, text) 1296 1297 return text 1298 1299 # "Sorta" because auto-links are identified as "tag" tokens. 1300 _sorta_html_tokenize_re = re.compile(r""" 1301 ( 1302 \\* # escapes 1303 (?: 1304 # tag 1305 </? 1306 (?:\w+) # tag name 1307 (?:\s+(?:[\w-]+:)?[\w-]+=(?:".*?"|'.*?'))* # attributes 1308 \s*/?> 1309 | 1310 # auto-link (e.g., <http://www.activestate.com/>) 1311 <[\w~:/?#\[\]@!$&'\(\)*+,;%=\.\\-]+> 1312 | 1313 <!--.*?--> # comment 1314 | 1315 <\?.*?\?> # processing instruction 1316 ) 1317 ) 1318 """, re.X) 1319 1320 def _escape_special_chars(self, text): 1321 # Python markdown note: the HTML tokenization here differs from 1322 # that in Markdown.pl, hence the behaviour for subtle cases can 1323 # differ (I believe the tokenizer here does a better job because 1324 # it isn't susceptible to unmatched '<' and '>' in HTML tags). 1325 # Note, however, that '>' is not allowed in an auto-link URL 1326 # here. 1327 lead_escape_re = re.compile(r'^((?:\\\\)*(?!\\))') 1328 escaped = [] 1329 is_html_markup = False 1330 for token in self._sorta_html_tokenize_re.split(text): 1331 # check token is preceded by 0 or more PAIRS of escapes, because escape pairs 1332 # escape themselves and don't affect the token 1333 if is_html_markup and lead_escape_re.match(token): 1334 # Within tags/HTML-comments/auto-links, encode * and _ 1335 # so they don't conflict with their use in Markdown for 1336 # italics and strong. We're replacing each such 1337 # character with its corresponding MD5 checksum value; 1338 # this is likely overkill, but it should prevent us from 1339 # colliding with the escape values by accident. 1340 escape_seq, token = lead_escape_re.split(token)[1:] or ('', token) 1341 escaped.append( 1342 escape_seq.replace('\\\\', self._escape_table['\\']) 1343 + token.replace('*', self._escape_table['*']) 1344 .replace('_', self._escape_table['_']) 1345 ) 1346 else: 1347 escaped.append(self._encode_backslash_escapes(token.replace('\\<', '<'))) 1348 is_html_markup = not is_html_markup 1349 return ''.join(escaped) 1350 1351 def _hash_html_spans(self, text): 1352 # Used for safe_mode. 1353 1354 def _is_auto_link(s): 1355 if ':' in s and self._auto_link_re.match(s): 1356 return True 1357 elif '@' in s and self._auto_email_link_re.match(s): 1358 return True 1359 return False 1360 1361 def _is_code_span(index, token): 1362 try: 1363 if token == '<code>': 1364 peek_tokens = split_tokens[index: index + 3] 1365 elif token == '</code>': 1366 peek_tokens = split_tokens[index - 2: index + 1] 1367 else: 1368 return False 1369 except IndexError: 1370 return False 1371 1372 return re.match(r'<code>md5-[A-Fa-f0-9]{32}</code>', ''.join(peek_tokens)) 1373 1374 tokens = [] 1375 split_tokens = self._sorta_html_tokenize_re.split(text) 1376 is_html_markup = False 1377 for index, token in enumerate(split_tokens): 1378 if is_html_markup and not _is_auto_link(token) and not _is_code_span(index, token): 1379 sanitized = self._sanitize_html(token) 1380 key = _hash_text(sanitized) 1381 self.html_spans[key] = sanitized 1382 tokens.append(key) 1383 else: 1384 tokens.append(self._encode_incomplete_tags(token)) 1385 is_html_markup = not is_html_markup 1386 return ''.join(tokens) 1387 1388 def _unhash_html_spans(self, text): 1389 for key, sanitized in list(self.html_spans.items()): 1390 text = text.replace(key, sanitized) 1391 return text 1392 1393 def _sanitize_html(self, s): 1394 if self.safe_mode == "replace": 1395 return self.html_removed_text 1396 elif self.safe_mode == "escape": 1397 replacements = [ 1398 ('&', '&'), 1399 ('<', '<'), 1400 ('>', '>'), 1401 ] 1402 for before, after in replacements: 1403 s = s.replace(before, after) 1404 return s 1405 else: 1406 raise MarkdownError("invalid value for 'safe_mode': %r (must be " 1407 "'escape' or 'replace')" % self.safe_mode) 1408 1409 _inline_link_title = re.compile(r''' 1410 ( # \1 1411 [ \t]+ 1412 (['"]) # quote char = \2 1413 (?P<title>.*?) 1414 \2 1415 )? # title is optional 1416 \)$ 1417 ''', re.X | re.S) 1418 _tail_of_reference_link_re = re.compile(r''' 1419 # Match tail of: [text][id] 1420 [ ]? # one optional space 1421 (?:\n[ ]*)? # one optional newline followed by spaces 1422 \[ 1423 (?P<id>.*?) 1424 \] 1425 ''', re.X | re.S) 1426 1427 _whitespace = re.compile(r'\s*') 1428 1429 _strip_anglebrackets = re.compile(r'<(.*)>.*') 1430 1431 def _find_non_whitespace(self, text, start): 1432 """Returns the index of the first non-whitespace character in text 1433 after (and including) start 1434 """ 1435 match = self._whitespace.match(text, start) 1436 return match.end() 1437 1438 def _find_balanced(self, text, start, open_c, close_c): 1439 """Returns the index where the open_c and close_c characters balance 1440 out - the same number of open_c and close_c are encountered - or the 1441 end of string if it's reached before the balance point is found. 1442 """ 1443 i = start 1444 l = len(text) 1445 count = 1 1446 while count > 0 and i < l: 1447 if text[i] == open_c: 1448 count += 1 1449 elif text[i] == close_c: 1450 count -= 1 1451 i += 1 1452 return i 1453 1454 def _extract_url_and_title(self, text, start): 1455 """Extracts the url and (optional) title from the tail of a link""" 1456 # text[start] equals the opening parenthesis 1457 idx = self._find_non_whitespace(text, start+1) 1458 if idx == len(text): 1459 return None, None, None 1460 end_idx = idx 1461 has_anglebrackets = text[idx] == "<" 1462 if has_anglebrackets: 1463 end_idx = self._find_balanced(text, end_idx+1, "<", ">") 1464 end_idx = self._find_balanced(text, end_idx, "(", ")") 1465 match = self._inline_link_title.search(text, idx, end_idx) 1466 if not match: 1467 return None, None, None 1468 url, title = text[idx:match.start()], match.group("title") 1469 if has_anglebrackets: 1470 url = self._strip_anglebrackets.sub(r'\1', url) 1471 return url, title, end_idx 1472 1473 def _protect_url(self, url): 1474 ''' 1475 Function that passes a URL through `_html_escape_url` to remove any nasty characters, 1476 and then hashes the now "safe" URL to prevent other safety mechanisms from tampering 1477 with it (eg: escaping "&" in URL parameters) 1478 ''' 1479 url = _html_escape_url(url, safe_mode=self.safe_mode) 1480 key = _hash_text(url) 1481 self._escape_table[url] = key 1482 return key 1483 1484 _safe_protocols = re.compile(r'(https?|ftp):', re.I) 1485 def _do_links(self, text): 1486 """Turn Markdown link shortcuts into XHTML <a> and <img> tags. 1487 1488 This is a combination of Markdown.pl's _DoAnchors() and 1489 _DoImages(). They are done together because that simplified the 1490 approach. It was necessary to use a different approach than 1491 Markdown.pl because of the lack of atomic matching support in 1492 Python's regex engine used in $g_nested_brackets. 1493 """ 1494 MAX_LINK_TEXT_SENTINEL = 3000 # markdown2 issue 24 1495 1496 # `anchor_allowed_pos` is used to support img links inside 1497 # anchors, but not anchors inside anchors. An anchor's start 1498 # pos must be `>= anchor_allowed_pos`. 1499 anchor_allowed_pos = 0 1500 1501 curr_pos = 0 1502 while True: # Handle the next link. 1503 # The next '[' is the start of: 1504 # - an inline anchor: [text](url "title") 1505 # - a reference anchor: [text][id] 1506 # - an inline img:  1507 # - a reference img: ![text][id] 1508 # - a footnote ref: [^id] 1509 # (Only if 'footnotes' extra enabled) 1510 # - a footnote defn: [^id]: ... 1511 # (Only if 'footnotes' extra enabled) These have already 1512 # been stripped in _strip_footnote_definitions() so no 1513 # need to watch for them. 1514 # - a link definition: [id]: url "title" 1515 # These have already been stripped in 1516 # _strip_link_definitions() so no need to watch for them. 1517 # - not markup: [...anything else... 1518 try: 1519 start_idx = text.index('[', curr_pos) 1520 except ValueError: 1521 break 1522 text_length = len(text) 1523 1524 # Find the matching closing ']'. 1525 # Markdown.pl allows *matching* brackets in link text so we 1526 # will here too. Markdown.pl *doesn't* currently allow 1527 # matching brackets in img alt text -- we'll differ in that 1528 # regard. 1529 bracket_depth = 0 1530 for p in range(start_idx+1, min(start_idx+MAX_LINK_TEXT_SENTINEL, 1531 text_length)): 1532 ch = text[p] 1533 if ch == ']': 1534 bracket_depth -= 1 1535 if bracket_depth < 0: 1536 break 1537 elif ch == '[': 1538 bracket_depth += 1 1539 else: 1540 # Closing bracket not found within sentinel length. 1541 # This isn't markup. 1542 curr_pos = start_idx + 1 1543 continue 1544 link_text = text[start_idx+1:p] 1545 1546 # Fix for issue 341 - Injecting XSS into link text 1547 if self.safe_mode: 1548 link_text = self._hash_html_spans(link_text) 1549 link_text = self._unhash_html_spans(link_text) 1550 1551 # Possibly a footnote ref? 1552 if "footnotes" in self.extras and link_text.startswith("^"): 1553 normed_id = re.sub(r'\W', '-', link_text[1:]) 1554 if normed_id in self.footnotes: 1555 self.footnote_ids.append(normed_id) 1556 result = '<sup class="footnote-ref" id="fnref-%s">' \ 1557 '<a href="#fn-%s">%s</a></sup>' \ 1558 % (normed_id, normed_id, len(self.footnote_ids)) 1559 text = text[:start_idx] + result + text[p+1:] 1560 else: 1561 # This id isn't defined, leave the markup alone. 1562 curr_pos = p+1 1563 continue 1564 1565 # Now determine what this is by the remainder. 1566 p += 1 1567 1568 # Inline anchor or img? 1569 if text[p:p + 1] == '(': # attempt at perf improvement 1570 url, title, url_end_idx = self._extract_url_and_title(text, p) 1571 if url is not None: 1572 # Handle an inline anchor or img. 1573 is_img = start_idx > 0 and text[start_idx-1] == "!" 1574 if is_img: 1575 start_idx -= 1 1576 1577 # We've got to encode these to avoid conflicting 1578 # with italics/bold. 1579 url = url.replace('*', self._escape_table['*']) \ 1580 .replace('_', self._escape_table['_']) 1581 if title: 1582 title_str = ' title="%s"' % ( 1583 _xml_escape_attr(title) 1584 .replace('*', self._escape_table['*']) 1585 .replace('_', self._escape_table['_'])) 1586 else: 1587 title_str = '' 1588 if is_img: 1589 img_class_str = self._html_class_str_from_tag("img") 1590 result = '<img src="%s" alt="%s"%s%s%s' \ 1591 % (self._protect_url(url), 1592 _xml_escape_attr(link_text), 1593 title_str, 1594 img_class_str, 1595 self.empty_element_suffix) 1596 if "smarty-pants" in self.extras: 1597 result = result.replace('"', self._escape_table['"']) 1598 curr_pos = start_idx + len(result) 1599 anchor_allowed_pos = start_idx + len(result) 1600 text = text[:start_idx] + result + text[url_end_idx:] 1601 elif start_idx >= anchor_allowed_pos: 1602 safe_link = self._safe_protocols.match(url) or url.startswith('#') 1603 if self.safe_mode and not safe_link: 1604 result_head = '<a href="#"%s>' % (title_str) 1605 else: 1606 result_head = '<a href="%s"%s>' % (self._protect_url(url), title_str) 1607 result = '%s%s</a>' % (result_head, link_text) 1608 if "smarty-pants" in self.extras: 1609 result = result.replace('"', self._escape_table['"']) 1610 # <img> allowed from curr_pos on, <a> from 1611 # anchor_allowed_pos on. 1612 curr_pos = start_idx + len(result_head) 1613 anchor_allowed_pos = start_idx + len(result) 1614 text = text[:start_idx] + result + text[url_end_idx:] 1615 else: 1616 # Anchor not allowed here. 1617 curr_pos = start_idx + 1 1618 continue 1619 1620 # Reference anchor or img? 1621 else: 1622 match = self._tail_of_reference_link_re.match(text, p) 1623 if match: 1624 # Handle a reference-style anchor or img. 1625 is_img = start_idx > 0 and text[start_idx-1] == "!" 1626 if is_img: 1627 start_idx -= 1 1628 link_id = match.group("id").lower() 1629 if not link_id: 1630 link_id = link_text.lower() # for links like [this][] 1631 if link_id in self.urls: 1632 url = self.urls[link_id] 1633 # We've got to encode these to avoid conflicting 1634 # with italics/bold. 1635 url = url.replace('*', self._escape_table['*']) \ 1636 .replace('_', self._escape_table['_']) 1637 title = self.titles.get(link_id) 1638 if title: 1639 title = _xml_escape_attr(title) \ 1640 .replace('*', self._escape_table['*']) \ 1641 .replace('_', self._escape_table['_']) 1642 title_str = ' title="%s"' % title 1643 else: 1644 title_str = '' 1645 if is_img: 1646 img_class_str = self._html_class_str_from_tag("img") 1647 result = '<img src="%s" alt="%s"%s%s%s' \ 1648 % (self._protect_url(url), 1649 _xml_escape_attr(link_text), 1650 title_str, 1651 img_class_str, 1652 self.empty_element_suffix) 1653 if "smarty-pants" in self.extras: 1654 result = result.replace('"', self._escape_table['"']) 1655 curr_pos = start_idx + len(result) 1656 text = text[:start_idx] + result + text[match.end():] 1657 elif start_idx >= anchor_allowed_pos: 1658 if self.safe_mode and not self._safe_protocols.match(url): 1659 result_head = '<a href="#"%s>' % (title_str) 1660 else: 1661 result_head = '<a href="%s"%s>' % (self._protect_url(url), title_str) 1662 result = '%s%s</a>' % (result_head, link_text) 1663 if "smarty-pants" in self.extras: 1664 result = result.replace('"', self._escape_table['"']) 1665 # <img> allowed from curr_pos on, <a> from 1666 # anchor_allowed_pos on. 1667 curr_pos = start_idx + len(result_head) 1668 anchor_allowed_pos = start_idx + len(result) 1669 text = text[:start_idx] + result + text[match.end():] 1670 else: 1671 # Anchor not allowed here. 1672 curr_pos = start_idx + 1 1673 else: 1674 # This id isn't defined, leave the markup alone. 1675 curr_pos = match.end() 1676 continue 1677 1678 # Otherwise, it isn't markup. 1679 curr_pos = start_idx + 1 1680 1681 return text 1682 1683 def header_id_from_text(self, text, prefix, n): 1684 """Generate a header id attribute value from the given header 1685 HTML content. 1686 1687 This is only called if the "header-ids" extra is enabled. 1688 Subclasses may override this for different header ids. 1689 1690 @param text {str} The text of the header tag 1691 @param prefix {str} The requested prefix for header ids. This is the 1692 value of the "header-ids" extra key, if any. Otherwise, None. 1693 @param n {int} The <hN> tag number, i.e. `1` for an <h1> tag. 1694 @returns {str} The value for the header tag's "id" attribute. Return 1695 None to not have an id attribute and to exclude this header from 1696 the TOC (if the "toc" extra is specified). 1697 """ 1698 header_id = _slugify(text) 1699 if prefix and isinstance(prefix, str): 1700 header_id = prefix + '-' + header_id 1701 1702 self._count_from_header_id[header_id] += 1 1703 if 0 == len(header_id) or self._count_from_header_id[header_id] > 1: 1704 header_id += '-%s' % self._count_from_header_id[header_id] 1705 1706 return header_id 1707 1708 def _toc_add_entry(self, level, id, name): 1709 if level > self._toc_depth: 1710 return 1711 if self._toc is None: 1712 self._toc = [] 1713 self._toc.append((level, id, self._unescape_special_chars(name))) 1714 1715 _h_re_base = r''' 1716 (^(.+)[ \t]{0,99}\n(=+|-+)[ \t]*\n+) 1717 | 1718 (^(\#{1,6}) # \1 = string of #'s 1719 [ \t]%s 1720 (.+?) # \2 = Header text 1721 [ \t]{0,99} 1722 (?<!\\) # ensure not an escaped trailing '#' 1723 \#* # optional closing #'s (not counted) 1724 \n+ 1725 ) 1726 ''' 1727 1728 _h_re = re.compile(_h_re_base % '*', re.X | re.M) 1729 _h_re_tag_friendly = re.compile(_h_re_base % '+', re.X | re.M) 1730 1731 def _h_sub(self, match): 1732 if match.group(1) is not None and match.group(3) == "-": 1733 return match.group(1) 1734 elif match.group(1) is not None: 1735 # Setext header 1736 n = {"=": 1, "-": 2}[match.group(3)[0]] 1737 header_group = match.group(2) 1738 else: 1739 # atx header 1740 n = len(match.group(5)) 1741 header_group = match.group(6) 1742 1743 demote_headers = self.extras.get("demote-headers") 1744 if demote_headers: 1745 n = min(n + demote_headers, 6) 1746 header_id_attr = "" 1747 if "header-ids" in self.extras: 1748 header_id = self.header_id_from_text(header_group, 1749 self.extras["header-ids"], n) 1750 if header_id: 1751 header_id_attr = ' id="%s"' % header_id 1752 html = self._run_span_gamut(header_group) 1753 if "toc" in self.extras and header_id: 1754 self._toc_add_entry(n, header_id, html) 1755 return "<h%d%s>%s</h%d>\n\n" % (n, header_id_attr, html, n) 1756 1757 def _do_headers(self, text): 1758 # Setext-style headers: 1759 # Header 1 1760 # ======== 1761 # 1762 # Header 2 1763 # -------- 1764 1765 # atx-style headers: 1766 # # Header 1 1767 # ## Header 2 1768 # ## Header 2 with closing hashes ## 1769 # ... 1770 # ###### Header 6 1771 1772 if 'tag-friendly' in self.extras: 1773 return self._h_re_tag_friendly.sub(self._h_sub, text) 1774 return self._h_re.sub(self._h_sub, text) 1775 1776 _marker_ul_chars = '*+-' 1777 _marker_any = r'(?:[%s]|\d+\.)' % _marker_ul_chars 1778 _marker_ul = '(?:[%s])' % _marker_ul_chars 1779 _marker_ol = r'(?:\d+\.)' 1780 1781 def _list_sub(self, match): 1782 lst = match.group(1) 1783 lst_type = match.group(4) in self._marker_ul_chars and "ul" or "ol" 1784 1785 if lst_type == 'ol' and match.group(4) != '1.': 1786 # if list doesn't start at 1 then set the ol start attribute 1787 lst_opts = ' start="%s"' % match.group(4)[:-1] 1788 else: 1789 lst_opts = '' 1790 1791 lst_opts = lst_opts + self._html_class_str_from_tag(lst_type) 1792 1793 result = self._process_list_items(lst) 1794 if self.list_level: 1795 return "<%s%s>\n%s</%s>\n" % (lst_type, lst_opts, result, lst_type) 1796 else: 1797 return "<%s%s>\n%s</%s>\n\n" % (lst_type, lst_opts, result, lst_type) 1798 1799 def _do_lists(self, text): 1800 # Form HTML ordered (numbered) and unordered (bulleted) lists. 1801 1802 # Iterate over each *non-overlapping* list match. 1803 pos = 0 1804 while True: 1805 # Find the *first* hit for either list style (ul or ol). We 1806 # match ul and ol separately to avoid adjacent lists of different 1807 # types running into each other (see issue #16). 1808 hits = [] 1809 for marker_pat in (self._marker_ul, self._marker_ol): 1810 less_than_tab = self.tab_width - 1 1811 other_marker_pat = self._marker_ul if marker_pat == self._marker_ol else self._marker_ol 1812 whole_list = r''' 1813 ( # \1 = whole list 1814 ( # \2 1815 ([ ]{0,%d}) # \3 = the indentation level of the list item marker 1816 (%s) # \4 = first list item marker 1817 [ \t]+ 1818 (?!\ *\4\ ) # '- - - ...' isn't a list. See 'not_quite_a_list' test case. 1819 ) 1820 (?:.+?) 1821 ( # \5 1822 \Z 1823 | 1824 \n{2,} 1825 (?=\S) 1826 (?! # Negative lookahead for another list item marker 1827 [ \t]* 1828 %s[ \t]+ 1829 ) 1830 | 1831 \n+ 1832 (?= 1833 \3 # lookahead for a different style of list item marker 1834 %s[ \t]+ 1835 ) 1836 ) 1837 ) 1838 ''' % (less_than_tab, marker_pat, marker_pat, other_marker_pat) 1839 if self.list_level: # sub-list 1840 list_re = re.compile("^"+whole_list, re.X | re.M | re.S) 1841 else: 1842 list_re = re.compile(r"(?:(?<=\n\n)|\A\n?)"+whole_list, 1843 re.X | re.M | re.S) 1844 match = list_re.search(text, pos) 1845 if match: 1846 hits.append((match.start(), match)) 1847 if not hits: 1848 break 1849 hits.sort() 1850 match = hits[0][1] 1851 start, end = match.span() 1852 middle = self._list_sub(match) 1853 text = text[:start] + middle + text[end:] 1854 pos = start + len(middle) # start pos for next attempted match 1855 1856 return text 1857 1858 _list_item_re = re.compile(r''' 1859 (\n)? # leading line = \1 1860 (^[ \t]*) # leading whitespace = \2 1861 (?P<marker>%s) [ \t]+ # list marker = \3 1862 ((?:.+?) # list item text = \4 1863 (\n{1,2})) # eols = \5 1864 (?= \n* (\Z | \2 (?P<next_marker>%s) [ \t]+)) 1865 ''' % (_marker_any, _marker_any), 1866 re.M | re.X | re.S) 1867 1868 _task_list_item_re = re.compile(r''' 1869 (\[[\ xX]\])[ \t]+ # tasklist marker = \1 1870 (.*) # list item text = \2 1871 ''', re.M | re.X | re.S) 1872 1873 _task_list_warpper_str = r'<input type="checkbox" class="task-list-item-checkbox" %sdisabled> %s' 1874 1875 def _task_list_item_sub(self, match): 1876 marker = match.group(1) 1877 item_text = match.group(2) 1878 if marker in ['[x]','[X]']: 1879 return self._task_list_warpper_str % ('checked ', item_text) 1880 elif marker == '[ ]': 1881 return self._task_list_warpper_str % ('', item_text) 1882 1883 _last_li_endswith_two_eols = False 1884 def _list_item_sub(self, match): 1885 item = match.group(4) 1886 leading_line = match.group(1) 1887 if leading_line or "\n\n" in item or self._last_li_endswith_two_eols: 1888 item = self._run_block_gamut(self._outdent(item)) 1889 else: 1890 # Recursion for sub-lists: 1891 item = self._do_lists(self._uniform_outdent(item, min_outdent=' ')[1]) 1892 if item.endswith('\n'): 1893 item = item[:-1] 1894 item = self._run_span_gamut(item) 1895 self._last_li_endswith_two_eols = (len(match.group(5)) == 2) 1896 1897 if "task_list" in self.extras: 1898 item = self._task_list_item_re.sub(self._task_list_item_sub, item) 1899 1900 return "<li>%s</li>\n" % item 1901 1902 def _process_list_items(self, list_str): 1903 # Process the contents of a single ordered or unordered list, 1904 # splitting it into individual list items. 1905 1906 # The $g_list_level global keeps track of when we're inside a list. 1907 # Each time we enter a list, we increment it; when we leave a list, 1908 # we decrement. If it's zero, we're not in a list anymore. 1909 # 1910 # We do this because when we're not inside a list, we want to treat 1911 # something like this: 1912 # 1913 # I recommend upgrading to version 1914 # 8. Oops, now this line is treated 1915 # as a sub-list. 1916 # 1917 # As a single paragraph, despite the fact that the second line starts 1918 # with a digit-period-space sequence. 1919 # 1920 # Whereas when we're inside a list (or sub-list), that line will be 1921 # treated as the start of a sub-list. What a kludge, huh? This is 1922 # an aspect of Markdown's syntax that's hard to parse perfectly 1923 # without resorting to mind-reading. Perhaps the solution is to 1924 # change the syntax rules such that sub-lists must start with a 1925 # starting cardinal number; e.g. "1." or "a.". 1926 self.list_level += 1 1927 self._last_li_endswith_two_eols = False 1928 list_str = list_str.rstrip('\n') + '\n' 1929 list_str = self._list_item_re.sub(self._list_item_sub, list_str) 1930 self.list_level -= 1 1931 return list_str 1932 1933 def _get_pygments_lexer(self, lexer_name): 1934 try: 1935 from pygments import lexers, util 1936 except ImportError: 1937 return None 1938 try: 1939 return lexers.get_lexer_by_name(lexer_name) 1940 except util.ClassNotFound: 1941 return None 1942 1943 def _color_with_pygments(self, codeblock, lexer, **formatter_opts): 1944 import pygments 1945 import pygments.formatters 1946 1947 class HtmlCodeFormatter(pygments.formatters.HtmlFormatter): 1948 def _wrap_code(self, inner): 1949 """A function for use in a Pygments Formatter which 1950 wraps in <code> tags. 1951 """ 1952 yield 0, "<code>" 1953 for tup in inner: 1954 yield tup 1955 yield 0, "</code>" 1956 1957 def _add_newline(self, inner): 1958 # Add newlines around the inner contents so that _strict_tag_block_re matches the outer div. 1959 yield 0, "\n" 1960 yield from inner 1961 yield 0, "\n" 1962 1963 def wrap(self, source, outfile=None): 1964 """Return the source with a code, pre, and div.""" 1965 if outfile is None: 1966 # pygments >= 2.12 1967 return self._add_newline(self._wrap_pre(self._wrap_code(source))) 1968 else: 1969 # pygments < 2.12 1970 return self._wrap_div(self._add_newline(self._wrap_pre(self._wrap_code(source)))) 1971 1972 formatter_opts.setdefault("cssclass", "codehilite") 1973 formatter = HtmlCodeFormatter(**formatter_opts) 1974 return pygments.highlight(codeblock, lexer, formatter) 1975 1976 def _code_block_sub(self, match, is_fenced_code_block=False): 1977 lexer_name = None 1978 if is_fenced_code_block: 1979 lexer_name = match.group(2) 1980 codeblock = match.group(3) 1981 codeblock = codeblock[:-1] # drop one trailing newline 1982 else: 1983 codeblock = match.group(1) 1984 codeblock = self._outdent(codeblock) 1985 codeblock = self._detab(codeblock) 1986 codeblock = codeblock.lstrip('\n') # trim leading newlines 1987 codeblock = codeblock.rstrip() # trim trailing whitespace 1988 1989 # Use pygments only if not using the highlightjs-lang extra 1990 if lexer_name and "highlightjs-lang" not in self.extras: 1991 lexer = self._get_pygments_lexer(lexer_name) 1992 if lexer: 1993 leading_indent = ' '*(len(match.group(1)) - len(match.group(1).lstrip())) 1994 return self._code_block_with_lexer_sub(codeblock, leading_indent, lexer, is_fenced_code_block) 1995 1996 pre_class_str = self._html_class_str_from_tag("pre") 1997 1998 if "highlightjs-lang" in self.extras and lexer_name: 1999 code_class_str = ' class="%s language-%s"' % (lexer_name, lexer_name) 2000 else: 2001 code_class_str = self._html_class_str_from_tag("code") 2002 2003 if is_fenced_code_block: 2004 # Fenced code blocks need to be outdented before encoding, and then reapplied 2005 leading_indent = ' ' * (len(match.group(1)) - len(match.group(1).lstrip())) 2006 if codeblock: 2007 # only run the codeblock through the outdenter if not empty 2008 leading_indent, codeblock = self._uniform_outdent(codeblock, max_outdent=leading_indent) 2009 2010 codeblock = self._encode_code(codeblock) 2011 2012 if lexer_name == 'mermaid' and 'mermaid' in self.extras: 2013 return '\n%s<pre class="mermaid-pre"><div class="mermaid">%s\n</div></pre>\n' % ( 2014 leading_indent, codeblock) 2015 2016 return "\n%s<pre%s><code%s>%s\n</code></pre>\n" % ( 2017 leading_indent, pre_class_str, code_class_str, codeblock) 2018 else: 2019 codeblock = self._encode_code(codeblock) 2020 2021 return "\n<pre%s><code%s>%s\n</code></pre>\n" % ( 2022 pre_class_str, code_class_str, codeblock) 2023 2024 def _code_block_with_lexer_sub(self, codeblock, leading_indent, lexer, is_fenced_code_block): 2025 if is_fenced_code_block: 2026 formatter_opts = self.extras['fenced-code-blocks'] or {} 2027 else: 2028 formatter_opts = {} 2029 2030 def unhash_code(codeblock): 2031 for key, sanitized in list(self.html_spans.items()): 2032 codeblock = codeblock.replace(key, sanitized) 2033 replacements = [ 2034 ("&", "&"), 2035 ("<", "<"), 2036 (">", ">") 2037 ] 2038 for old, new in replacements: 2039 codeblock = codeblock.replace(old, new) 2040 return codeblock 2041 # remove leading indent from code block 2042 _, codeblock = self._uniform_outdent(codeblock, max_outdent=leading_indent) 2043 2044 codeblock = unhash_code(codeblock) 2045 colored = self._color_with_pygments(codeblock, lexer, 2046 **formatter_opts) 2047 2048 # add back the indent to all lines 2049 return "\n%s\n" % self._uniform_indent(colored, leading_indent, True) 2050 2051 def _html_class_str_from_tag(self, tag): 2052 """Get the appropriate ' class="..."' string (note the leading 2053 space), if any, for the given tag. 2054 """ 2055 if "html-classes" not in self.extras: 2056 return "" 2057 try: 2058 html_classes_from_tag = self.extras["html-classes"] 2059 except TypeError: 2060 return "" 2061 else: 2062 if isinstance(html_classes_from_tag, dict): 2063 if tag in html_classes_from_tag: 2064 return ' class="%s"' % html_classes_from_tag[tag] 2065 return "" 2066 2067 def _do_code_blocks(self, text): 2068 """Process Markdown `<pre><code>` blocks.""" 2069 code_block_re = re.compile(r''' 2070 (?:\n\n|\A\n?) 2071 ( # $1 = the code block -- one or more lines, starting with a space/tab 2072 (?: 2073 (?:[ ]{%d} | \t) # Lines must start with a tab or a tab-width of spaces 2074 .*\n+ 2075 )+ 2076 ) 2077 ((?=^[ ]{0,%d}\S)|\Z) # Lookahead for non-space at line-start, or end of doc 2078 # Lookahead to make sure this block isn't already in a code block. 2079 # Needed when syntax highlighting is being used. 2080 (?!([^<]|<(/?)span)*\</code\>) 2081 ''' % (self.tab_width, self.tab_width), 2082 re.M | re.X) 2083 return code_block_re.sub(self._code_block_sub, text) 2084 2085 _fenced_code_block_re = re.compile(r''' 2086 (?:\n+|\A\n?|(?<=\n)) 2087 (^[ \t]*`{3,})\s{0,99}?([\w+-]+)?\s{0,99}?\n # $1 = opening fence (captured for back-referencing), $2 = optional lang 2088 (.*?) # $3 = code block content 2089 \1[ \t]*\n # closing fence 2090 ''', re.M | re.X | re.S) 2091 2092 def _fenced_code_block_sub(self, match): 2093 return self._code_block_sub(match, is_fenced_code_block=True) 2094 2095 def _do_fenced_code_blocks(self, text): 2096 """Process ```-fenced unindented code blocks ('fenced-code-blocks' extra).""" 2097 return self._fenced_code_block_re.sub(self._fenced_code_block_sub, text) 2098 2099 # Rules for a code span: 2100 # - backslash escapes are not interpreted in a code span 2101 # - to include one or or a run of more backticks the delimiters must 2102 # be a longer run of backticks 2103 # - cannot start or end a code span with a backtick; pad with a 2104 # space and that space will be removed in the emitted HTML 2105 # See `test/tm-cases/escapes.text` for a number of edge-case 2106 # examples. 2107 _code_span_re = re.compile(r''' 2108 (?<!\\) 2109 (`+) # \1 = Opening run of ` 2110 (?!`) # See Note A test/tm-cases/escapes.text 2111 (.+?) # \2 = The code block 2112 (?<!`) 2113 \1 # Matching closer 2114 (?!`) 2115 ''', re.X | re.S) 2116 2117 def _code_span_sub(self, match): 2118 c = match.group(2).strip(" \t") 2119 c = self._encode_code(c) 2120 return "<code%s>%s</code>" % (self._html_class_str_from_tag("code"), c) 2121 2122 def _do_code_spans(self, text): 2123 # * Backtick quotes are used for <code></code> spans. 2124 # 2125 # * You can use multiple backticks as the delimiters if you want to 2126 # include literal backticks in the code span. So, this input: 2127 # 2128 # Just type ``foo `bar` baz`` at the prompt. 2129 # 2130 # Will translate to: 2131 # 2132 # <p>Just type <code>foo `bar` baz</code> at the prompt.</p> 2133 # 2134 # There's no arbitrary limit to the number of backticks you 2135 # can use as delimters. If you need three consecutive backticks 2136 # in your code, use four for delimiters, etc. 2137 # 2138 # * You can use spaces to get literal backticks at the edges: 2139 # 2140 # ... type `` `bar` `` ... 2141 # 2142 # Turns to: 2143 # 2144 # ... type <code>`bar`</code> ... 2145 return self._code_span_re.sub(self._code_span_sub, text) 2146 2147 def _encode_code(self, text): 2148 """Encode/escape certain characters inside Markdown code runs. 2149 The point is that in code, these characters are literals, 2150 and lose their special Markdown meanings. 2151 """ 2152 replacements = [ 2153 # Encode all ampersands; HTML entities are not 2154 # entities within a Markdown code span. 2155 ('&', '&'), 2156 # Do the angle bracket song and dance: 2157 ('<', '<'), 2158 ('>', '>'), 2159 ] 2160 for before, after in replacements: 2161 text = text.replace(before, after) 2162 hashed = _hash_text(text) 2163 self._code_table[text] = hashed 2164 return hashed 2165 2166 def _wavedrom_block_sub(self, match): 2167 # if this isn't a wavedrom diagram block, exit now 2168 if match.group(2) != 'wavedrom': 2169 return match.string[match.start():match.end()] 2170 2171 # dedent the block for processing 2172 lead_indent, waves = self._uniform_outdent(match.group(3)) 2173 # default tags to wrap the wavedrom block in 2174 open_tag, close_tag = '<script type="WaveDrom">\n', '</script>' 2175 2176 # check if the user would prefer to have the SVG embedded directly 2177 if not isinstance(self.extras['wavedrom'], dict): 2178 embed_svg = True 2179 else: 2180 # default behaviour is to embed SVGs 2181 embed_svg = self.extras['wavedrom'].get('prefer_embed_svg', True) 2182 2183 if embed_svg: 2184 try: 2185 import wavedrom 2186 waves = wavedrom.render(waves).tostring() 2187 open_tag, close_tag = '<div>', '\n</div>' 2188 except ImportError: 2189 pass 2190 2191 # hash SVG to prevent <> chars being messed with 2192 self._escape_table[waves] = _hash_text(waves) 2193 2194 return self._uniform_indent( 2195 '\n%s%s%s\n' % (open_tag, self._escape_table[waves], close_tag), 2196 lead_indent, include_empty_lines=True 2197 ) 2198 2199 def _do_wavedrom_blocks(self, text): 2200 return self._fenced_code_block_re.sub(self._wavedrom_block_sub, text) 2201 2202 _admonitions = r'admonition|attention|caution|danger|error|hint|important|note|tip|warning' 2203 _admonitions_re = re.compile(r''' 2204 ^(\ *)\.\.\ (%s)::\ * # $1 leading indent, $2 the admonition 2205 (.*)? # $3 admonition title 2206 ((?:\s*\n\1\ {3,}.*)+?) # $4 admonition body (required) 2207 (?=\s*(?:\Z|\n{4,}|\n\1?\ {0,2}\S)) # until EOF, 3 blank lines or something less indented 2208 ''' % _admonitions, 2209 re.IGNORECASE | re.MULTILINE | re.VERBOSE 2210 ) 2211 2212 def _do_admonitions_sub(self, match): 2213 lead_indent, admonition_name, title, body = match.groups() 2214 2215 admonition_type = '<strong>%s</strong>' % admonition_name 2216 2217 # figure out the class names to assign the block 2218 if admonition_name.lower() == 'admonition': 2219 admonition_class = 'admonition' 2220 else: 2221 admonition_class = 'admonition %s' % admonition_name.lower() 2222 2223 # titles are generally optional 2224 if title: 2225 title = '<em>%s</em>' % title 2226 2227 # process the admonition body like regular markdown 2228 body = self._run_block_gamut("\n%s\n" % self._uniform_outdent(body)[1]) 2229 2230 # indent the body before placing inside the aside block 2231 admonition = self._uniform_indent('%s\n%s\n\n%s\n' % (admonition_type, title, body), self.tab, False) 2232 # wrap it in an aside 2233 admonition = '<aside class="%s">\n%s</aside>' % (admonition_class, admonition) 2234 # now indent the whole admonition back to where it started 2235 return self._uniform_indent(admonition, lead_indent, False) 2236 2237 def _do_admonitions(self, text): 2238 return self._admonitions_re.sub(self._do_admonitions_sub, text) 2239 2240 _strike_re = re.compile(r"~~(?=\S)(.+?)(?<=\S)~~", re.S) 2241 def _do_strike(self, text): 2242 text = self._strike_re.sub(r"<s>\1</s>", text) 2243 return text 2244 2245 _underline_re = re.compile(r"(?<!<!)--(?!>)(?=\S)(.+?)(?<=\S)(?<!<!)--(?!>)", re.S) 2246 def _do_underline(self, text): 2247 text = self._underline_re.sub(r"<u>\1</u>", text) 2248 return text 2249 2250 _tg_spoiler_re = re.compile(r"\|\|\s?(.+?)\s?\|\|", re.S) 2251 def _do_tg_spoiler(self, text): 2252 text = self._tg_spoiler_re.sub(r"<tg-spoiler>\1</tg-spoiler>", text) 2253 return text 2254 2255 _strong_re = re.compile(r"(\*\*|__)(?=\S)(.+?[*_]*)(?<=\S)\1", re.S) 2256 _em_re = re.compile(r"(\*|_)(?=\S)(.+?)(?<=\S)\1", re.S) 2257 _code_friendly_strong_re = re.compile(r"\*\*(?=\S)(.+?[*_]*)(?<=\S)\*\*", re.S) 2258 _code_friendly_em_re = re.compile(r"\*(?=\S)(.+?)(?<=\S)\*", re.S) 2259 def _do_italics_and_bold(self, text): 2260 # <strong> must go first: 2261 if "code-friendly" in self.extras: 2262 text = self._code_friendly_strong_re.sub(r"<strong>\1</strong>", text) 2263 text = self._code_friendly_em_re.sub(r"<em>\1</em>", text) 2264 else: 2265 text = self._strong_re.sub(r"<strong>\2</strong>", text) 2266 text = self._em_re.sub(r"<em>\2</em>", text) 2267 return text 2268 2269 # "smarty-pants" extra: Very liberal in interpreting a single prime as an 2270 # apostrophe; e.g. ignores the fact that "round", "bout", "twer", and 2271 # "twixt" can be written without an initial apostrophe. This is fine because 2272 # using scare quotes (single quotation marks) is rare. 2273 _apostrophe_year_re = re.compile(r"'(\d\d)(?=(\s|,|;|\.|\?|!|$))") 2274 _contractions = ["tis", "twas", "twer", "neath", "o", "n", 2275 "round", "bout", "twixt", "nuff", "fraid", "sup"] 2276 def _do_smart_contractions(self, text): 2277 text = self._apostrophe_year_re.sub(r"’\1", text) 2278 for c in self._contractions: 2279 text = text.replace("'%s" % c, "’%s" % c) 2280 text = text.replace("'%s" % c.capitalize(), 2281 "’%s" % c.capitalize()) 2282 return text 2283 2284 # Substitute double-quotes before single-quotes. 2285 _opening_single_quote_re = re.compile(r"(?<!\S)'(?=\S)") 2286 _opening_double_quote_re = re.compile(r'(?<!\S)"(?=\S)') 2287 _closing_single_quote_re = re.compile(r"(?<=\S)'") 2288 _closing_double_quote_re = re.compile(r'(?<=\S)"(?=(\s|,|;|\.|\?|!|$))') 2289 def _do_smart_punctuation(self, text): 2290 """Fancifies 'single quotes', "double quotes", and apostrophes. 2291 Converts --, ---, and ... into en dashes, em dashes, and ellipses. 2292 2293 Inspiration is: <http://daringfireball.net/projects/smartypants/> 2294 See "test/tm-cases/smarty_pants.text" for a full discussion of the 2295 support here and 2296 <http://code.google.com/p/python-markdown2/issues/detail?id=42> for a 2297 discussion of some diversion from the original SmartyPants. 2298 """ 2299 if "'" in text: # guard for perf 2300 text = self._do_smart_contractions(text) 2301 text = self._opening_single_quote_re.sub("‘", text) 2302 text = self._closing_single_quote_re.sub("’", text) 2303 2304 if '"' in text: # guard for perf 2305 text = self._opening_double_quote_re.sub("“", text) 2306 text = self._closing_double_quote_re.sub("”", text) 2307 2308 text = text.replace("---", "—") 2309 text = text.replace("--", "–") 2310 text = text.replace("...", "…") 2311 text = text.replace(" . . . ", "…") 2312 text = text.replace(". . .", "…") 2313 2314 # TODO: Temporary hack to fix https://github.com/trentm/python-markdown2/issues/150 2315 if "footnotes" in self.extras and "footnote-ref" in text: 2316 # Quotes in the footnote back ref get converted to "smart" quotes 2317 # Change them back here to ensure they work. 2318 text = text.replace('class="footnote-ref”', 'class="footnote-ref"') 2319 2320 return text 2321 2322 _block_quote_base = r''' 2323 ( # Wrap whole match in \1 2324 ( 2325 ^[ \t]*>%s[ \t]? # '>' at the start of a line 2326 .+\n # rest of the first line 2327 (.+\n)* # subsequent consecutive lines 2328 )+ 2329 ) 2330 ''' 2331 _block_quote_re = re.compile(_block_quote_base % '', re.M | re.X) 2332 _block_quote_re_spoiler = re.compile(_block_quote_base % '[ \t]*?!?', re.M | re.X) 2333 _bq_one_level_re = re.compile('^[ \t]*>[ \t]?', re.M) 2334 _bq_one_level_re_spoiler = re.compile('^[ \t]*>[ \t]*?![ \t]?', re.M) 2335 _bq_all_lines_spoilers = re.compile(r'\A(?:^[ \t]*>[ \t]*?!.*[\n\r]*)+\Z', re.M) 2336 _html_pre_block_re = re.compile(r'(\s*<pre>.+?</pre>)', re.S) 2337 def _dedent_two_spaces_sub(self, match): 2338 return re.sub(r'(?m)^ ', '', match.group(1)) 2339 2340 def _block_quote_sub(self, match): 2341 bq = match.group(1) 2342 is_spoiler = 'spoiler' in self.extras and self._bq_all_lines_spoilers.match(bq) 2343 # trim one level of quoting 2344 if is_spoiler: 2345 bq = self._bq_one_level_re_spoiler.sub('', bq) 2346 else: 2347 bq = self._bq_one_level_re.sub('', bq) 2348 # trim whitespace-only lines 2349 bq = self._ws_only_line_re.sub('', bq) 2350 bq = self._run_block_gamut(bq) # recurse 2351 2352 bq = re.sub('(?m)^', ' ', bq) 2353 # These leading spaces screw with <pre> content, so we need to fix that: 2354 bq = self._html_pre_block_re.sub(self._dedent_two_spaces_sub, bq) 2355 2356 if is_spoiler: 2357 return '<blockquote class="spoiler">\n%s\n</blockquote>\n\n' % bq 2358 else: 2359 return '<blockquote>\n%s\n</blockquote>\n\n' % bq 2360 2361 def _do_block_quotes(self, text): 2362 if '>' not in text: 2363 return text 2364 if 'spoiler' in self.extras: 2365 return self._block_quote_re_spoiler.sub(self._block_quote_sub, text) 2366 else: 2367 return self._block_quote_re.sub(self._block_quote_sub, text) 2368 2369 def _form_paragraphs(self, text): 2370 # Strip leading and trailing lines: 2371 text = text.strip('\n') 2372 2373 # Wrap <p> tags. 2374 grafs = [] 2375 for i, graf in enumerate(re.split(r"\n{2,}", text)): 2376 if graf in self.html_blocks: 2377 # Unhashify HTML blocks 2378 grafs.append(self.html_blocks[graf]) 2379 else: 2380 cuddled_list = None 2381 if "cuddled-lists" in self.extras: 2382 # Need to put back trailing '\n' for `_list_item_re` 2383 # match at the end of the paragraph. 2384 li = self._list_item_re.search(graf + '\n') 2385 # Two of the same list marker in this paragraph: a likely 2386 # candidate for a list cuddled to preceding paragraph 2387 # text (issue 33). Note the `[-1]` is a quick way to 2388 # consider numeric bullets (e.g. "1." and "2.") to be 2389 # equal. 2390 if (li and len(li.group(2)) <= 3 2391 and ( 2392 (li.group("next_marker") and li.group("marker")[-1] == li.group("next_marker")[-1]) 2393 or 2394 li.group("next_marker") is None 2395 ) 2396 ): 2397 start = li.start() 2398 cuddled_list = self._do_lists(graf[start:]).rstrip("\n") 2399 assert re.match(r'^<(?:ul|ol).*?>', cuddled_list) 2400 graf = graf[:start] 2401 2402 # Wrap <p> tags. 2403 graf = self._run_span_gamut(graf) 2404 grafs.append("<p%s>" % self._html_class_str_from_tag('p') + graf.lstrip(" \t") + "</p>") 2405 2406 if cuddled_list: 2407 grafs.append(cuddled_list) 2408 2409 return "\n\n".join(grafs) 2410 2411 def _add_footnotes(self, text): 2412 if self.footnotes: 2413 footer = [ 2414 '<div class="footnotes">', 2415 '<hr' + self.empty_element_suffix, 2416 '<ol>', 2417 ] 2418 2419 if not self.footnote_title: 2420 self.footnote_title = "Jump back to footnote %d in the text." 2421 if not self.footnote_return_symbol: 2422 self.footnote_return_symbol = "↩" 2423 2424 for i, id in enumerate(self.footnote_ids): 2425 if i != 0: 2426 footer.append('') 2427 footer.append('<li id="fn-%s">' % id) 2428 footer.append(self._run_block_gamut(self.footnotes[id])) 2429 try: 2430 backlink = ('<a href="#fnref-%s" ' + 2431 'class="footnoteBackLink" ' + 2432 'title="' + self.footnote_title + '">' + 2433 self.footnote_return_symbol + 2434 '</a>') % (id, i+1) 2435 except TypeError: 2436 log.debug("Footnote error. `footnote_title` " 2437 "must include parameter. Using defaults.") 2438 backlink = ('<a href="#fnref-%s" ' 2439 'class="footnoteBackLink" ' 2440 'title="Jump back to footnote %d in the text.">' 2441 '↩</a>' % (id, i+1)) 2442 2443 if footer[-1].endswith("</p>"): 2444 footer[-1] = footer[-1][:-len("</p>")] \ 2445 + ' ' + backlink + "</p>" 2446 else: 2447 footer.append("\n<p>%s</p>" % backlink) 2448 footer.append('</li>') 2449 footer.append('</ol>') 2450 footer.append('</div>') 2451 return text + '\n\n' + '\n'.join(footer) 2452 else: 2453 return text 2454 2455 _naked_lt_re = re.compile(r'<(?![a-z/?\$!])', re.I) 2456 _naked_gt_re = re.compile(r'''(?<![a-z0-9?!/'"-])>''', re.I) 2457 2458 def _encode_amps_and_angles(self, text): 2459 # Smart processing for ampersands and angle brackets that need 2460 # to be encoded. 2461 text = _AMPERSAND_RE.sub('&', text) 2462 2463 # Encode naked <'s 2464 text = self._naked_lt_re.sub('<', text) 2465 2466 # Encode naked >'s 2467 # Note: Other markdown implementations (e.g. Markdown.pl, PHP 2468 # Markdown) don't do this. 2469 text = self._naked_gt_re.sub('>', text) 2470 return text 2471 2472 _incomplete_tags_re = re.compile(r"<(/?\w+?(?!\w)\s*?.+?[\s/]+?)") 2473 2474 def _encode_incomplete_tags(self, text): 2475 if self.safe_mode not in ("replace", "escape"): 2476 return text 2477 2478 if text.endswith(">"): 2479 return text # this is not an incomplete tag, this is a link in the form <http://x.y.z> 2480 2481 def incomplete_tags_sub(match): 2482 return match.group().replace('<', '<') 2483 2484 return self._incomplete_tags_re.sub(incomplete_tags_sub, text) 2485 2486 def _encode_backslash_escapes(self, text): 2487 for ch, escape in list(self._escape_table.items()): 2488 text = text.replace("\\"+ch, escape) 2489 return text 2490 2491 _auto_link_re = re.compile(r'<((https?|ftp):[^\'">\s]+)>', re.I) 2492 def _auto_link_sub(self, match): 2493 g1 = match.group(1) 2494 return '<a href="%s">%s</a>' % (self._protect_url(g1), g1) 2495 2496 _auto_email_link_re = re.compile(r""" 2497 < 2498 (?:mailto:)? 2499 ( 2500 [-.\w]+ 2501 \@ 2502 [-\w]+(\.[-\w]+)*\.[a-z]+ 2503 ) 2504 > 2505 """, re.I | re.X | re.U) 2506 def _auto_email_link_sub(self, match): 2507 return self._encode_email_address( 2508 self._unescape_special_chars(match.group(1))) 2509 2510 def _do_auto_links(self, text): 2511 text = self._auto_link_re.sub(self._auto_link_sub, text) 2512 text = self._auto_email_link_re.sub(self._auto_email_link_sub, text) 2513 return text 2514 2515 def _encode_email_address(self, addr): 2516 # Input: an email address, e.g. "foo@example.com" 2517 # 2518 # Output: the email address as a mailto link, with each character 2519 # of the address encoded as either a decimal or hex entity, in 2520 # the hopes of foiling most address harvesting spam bots. E.g.: 2521 # 2522 # <a href="mailto:foo@e 2523 # xample.com">foo 2524 # @example.com</a> 2525 # 2526 # Based on a filter by Matthew Wickline, posted to the BBEdit-Talk 2527 # mailing list: <http://tinyurl.com/yu7ue> 2528 chars = [_xml_encode_email_char_at_random(ch) 2529 for ch in "mailto:" + addr] 2530 # Strip the mailto: from the visible part. 2531 addr = '<a href="%s">%s</a>' \ 2532 % (''.join(chars), ''.join(chars[7:])) 2533 return addr 2534 2535 _basic_link_re = re.compile(r'!?\[.*?\]\(.*?\)') 2536 def _do_link_patterns(self, text): 2537 link_from_hash = {} 2538 for regex, repl in self.link_patterns: 2539 replacements = [] 2540 for match in regex.finditer(text): 2541 if any(self._match_overlaps_substr(text, match, h) for h in link_from_hash): 2542 continue 2543 2544 if hasattr(repl, "__call__"): 2545 href = repl(match) 2546 else: 2547 href = match.expand(repl) 2548 replacements.append((match.span(), href)) 2549 for (start, end), href in reversed(replacements): 2550 2551 # Do not match against links inside brackets. 2552 if text[start - 1:start] == '[' and text[end:end + 1] == ']': 2553 continue 2554 2555 # Do not match against links in the standard markdown syntax. 2556 if text[start - 2:start] == '](' or text[end:end + 2] == '")': 2557 continue 2558 2559 # Do not match against links which are escaped. 2560 if text[start - 3:start] == '"""' and text[end:end + 3] == '"""': 2561 text = text[:start - 3] + text[start:end] + text[end + 3:] 2562 continue 2563 2564 # search the text for anything that looks like a link 2565 is_inside_link = False 2566 for link_re in (self._auto_link_re, self._basic_link_re): 2567 for match in link_re.finditer(text): 2568 if any((r[0] <= start and end <= r[1]) for r in match.regs): 2569 # if the link pattern start and end pos is within the bounds of 2570 # something that looks like a link, then don't process it 2571 is_inside_link = True 2572 break 2573 else: 2574 continue 2575 break 2576 2577 if is_inside_link: 2578 continue 2579 2580 escaped_href = ( 2581 href.replace('"', '"') # b/c of attr quote 2582 # To avoid markdown <em> and <strong>: 2583 .replace('*', self._escape_table['*']) 2584 .replace('_', self._escape_table['_'])) 2585 link = '<a href="%s">%s</a>' % (escaped_href, text[start:end]) 2586 hash = _hash_text(link) 2587 link_from_hash[hash] = link 2588 text = text[:start] + hash + text[end:] 2589 for hash, link in list(link_from_hash.items()): 2590 text = text.replace(hash, link) 2591 return text 2592 2593 def _unescape_special_chars(self, text): 2594 # Swap back in all the special characters we've hidden. 2595 while True: 2596 orig_text = text 2597 for ch, hash in list(self._escape_table.items()) + list(self._code_table.items()): 2598 text = text.replace(hash, ch) 2599 if text == orig_text: 2600 break 2601 return text 2602 2603 def _outdent(self, text): 2604 # Remove one level of line-leading tabs or spaces 2605 return self._outdent_re.sub('', text) 2606 2607 def _uniform_outdent(self, text, min_outdent=None, max_outdent=None): 2608 # Removes the smallest common leading indentation from each (non empty) 2609 # line of `text` and returns said indent along with the outdented text. 2610 # The `min_outdent` kwarg makes sure the smallest common whitespace 2611 # must be at least this size 2612 # The `max_outdent` sets the maximum amount a line can be 2613 # outdented by 2614 2615 # find the leading whitespace for every line 2616 whitespace = [ 2617 re.findall(r'^[ \t]*', line)[0] if line else None 2618 for line in text.splitlines() 2619 ] 2620 whitespace_not_empty = [i for i in whitespace if i is not None] 2621 2622 # if no whitespace detected (ie: no lines in code block, issue #505) 2623 if not whitespace_not_empty: 2624 return '', text 2625 2626 # get minimum common whitespace 2627 outdent = min(whitespace_not_empty) 2628 # adjust min common ws to be within bounds 2629 if min_outdent is not None: 2630 outdent = min([i for i in whitespace_not_empty if i >= min_outdent] or [min_outdent]) 2631 if max_outdent is not None: 2632 outdent = min(outdent, max_outdent) 2633 2634 outdented = [] 2635 for line_ws, line in zip(whitespace, text.splitlines(True)): 2636 if line.startswith(outdent): 2637 # if line starts with smallest common ws, dedent it 2638 outdented.append(line.replace(outdent, '', 1)) 2639 elif line_ws is not None and line_ws < outdent: 2640 # if less indented than min common whitespace then outdent as much as possible 2641 outdented.append(line.replace(line_ws, '', 1)) 2642 else: 2643 outdented.append(line) 2644 2645 return outdent, ''.join(outdented) 2646 2647 def _uniform_indent(self, text, indent, include_empty_lines=False): 2648 return ''.join( 2649 (indent + line if line.strip() or include_empty_lines else '') 2650 for line in text.splitlines(True) 2651 ) 2652 2653 @staticmethod 2654 def _match_overlaps_substr(text, match, substr): 2655 ''' 2656 Checks if a regex match overlaps with a substring in the given text. 2657 ''' 2658 for instance in re.finditer(re.escape(substr), text): 2659 start, end = instance.span() 2660 if start <= match.start() <= end: 2661 return True 2662 if start <= match.end() <= end: 2663 return True 2664 return False 2665 2666 2667class MarkdownWithExtras(Markdown): 2668 """A markdowner class that enables most extras: 2669 2670 - footnotes 2671 - fenced-code-blocks (only highlights code if 'pygments' Python module on path) 2672 2673 These are not included: 2674 - pyshell (specific to Python-related documenting) 2675 - code-friendly (because it *disables* part of the syntax) 2676 - link-patterns (because you need to specify some actual 2677 link-patterns anyway) 2678 """ 2679 extras = ["footnotes", "fenced-code-blocks"] 2680 2681 2682# ---- internal support functions 2683 2684 2685def calculate_toc_html(toc): 2686 """Return the HTML for the current TOC. 2687 2688 This expects the `_toc` attribute to have been set on this instance. 2689 """ 2690 if toc is None: 2691 return None 2692 2693 def indent(): 2694 return ' ' * (len(h_stack) - 1) 2695 lines = [] 2696 h_stack = [0] # stack of header-level numbers 2697 for level, id, name in toc: 2698 if level > h_stack[-1]: 2699 lines.append("%s<ul>" % indent()) 2700 h_stack.append(level) 2701 elif level == h_stack[-1]: 2702 lines[-1] += "</li>" 2703 else: 2704 while level < h_stack[-1]: 2705 h_stack.pop() 2706 if not lines[-1].endswith("</li>"): 2707 lines[-1] += "</li>" 2708 lines.append("%s</ul></li>" % indent()) 2709 lines.append('%s<li><a href="#%s">%s</a>' % ( 2710 indent(), id, name)) 2711 while len(h_stack) > 1: 2712 h_stack.pop() 2713 if not lines[-1].endswith("</li>"): 2714 lines[-1] += "</li>" 2715 lines.append("%s</ul>" % indent()) 2716 return '\n'.join(lines) + '\n' 2717 2718 2719class UnicodeWithAttrs(str): 2720 """A subclass of unicode used for the return value of conversion to 2721 possibly attach some attributes. E.g. the "toc_html" attribute when 2722 the "toc" extra is used. 2723 """ 2724 metadata = None 2725 toc_html = None 2726 2727## {{{ http://code.activestate.com/recipes/577257/ (r1) 2728_slugify_strip_re = re.compile(r'[^\w\s-]') 2729_slugify_hyphenate_re = re.compile(r'[-\s]+') 2730def _slugify(value): 2731 """ 2732 Normalizes string, converts to lowercase, removes non-alpha characters, 2733 and converts spaces to hyphens. 2734 2735 From Django's "django/template/defaultfilters.py". 2736 """ 2737 import unicodedata 2738 value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore').decode() 2739 value = _slugify_strip_re.sub('', value).strip().lower() 2740 return _slugify_hyphenate_re.sub('-', value) 2741## end of http://code.activestate.com/recipes/577257/ }}} 2742 2743 2744# From http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52549 2745def _curry(*args, **kwargs): 2746 function, args = args[0], args[1:] 2747 def result(*rest, **kwrest): 2748 combined = kwargs.copy() 2749 combined.update(kwrest) 2750 return function(*args + rest, **combined) 2751 return result 2752 2753 2754# Recipe: regex_from_encoded_pattern (1.0) 2755def _regex_from_encoded_pattern(s): 2756 """'foo' -> re.compile(re.escape('foo')) 2757 '/foo/' -> re.compile('foo') 2758 '/foo/i' -> re.compile('foo', re.I) 2759 """ 2760 if s.startswith('/') and s.rfind('/') != 0: 2761 # Parse it: /PATTERN/FLAGS 2762 idx = s.rfind('/') 2763 _, flags_str = s[1:idx], s[idx+1:] 2764 flag_from_char = { 2765 "i": re.IGNORECASE, 2766 "l": re.LOCALE, 2767 "s": re.DOTALL, 2768 "m": re.MULTILINE, 2769 "u": re.UNICODE, 2770 } 2771 flags = 0 2772 for char in flags_str: 2773 try: 2774 flags |= flag_from_char[char] 2775 except KeyError: 2776 raise ValueError("unsupported regex flag: '%s' in '%s' " 2777 "(must be one of '%s')" 2778 % (char, s, ''.join(list(flag_from_char.keys())))) 2779 return re.compile(s[1:idx], flags) 2780 else: # not an encoded regex 2781 return re.compile(re.escape(s)) 2782 2783 2784# Recipe: dedent (0.1.2) 2785def _dedentlines(lines, tabsize=8, skip_first_line=False): 2786 """_dedentlines(lines, tabsize=8, skip_first_line=False) -> dedented lines 2787 2788 "lines" is a list of lines to dedent. 2789 "tabsize" is the tab width to use for indent width calculations. 2790 "skip_first_line" is a boolean indicating if the first line should 2791 be skipped for calculating the indent width and for dedenting. 2792 This is sometimes useful for docstrings and similar. 2793 2794 Same as dedent() except operates on a sequence of lines. Note: the 2795 lines list is modified **in-place**. 2796 """ 2797 DEBUG = False 2798 if DEBUG: 2799 print("dedent: dedent(..., tabsize=%d, skip_first_line=%r)"\ 2800 % (tabsize, skip_first_line)) 2801 margin = None 2802 for i, line in enumerate(lines): 2803 if i == 0 and skip_first_line: continue 2804 indent = 0 2805 for ch in line: 2806 if ch == ' ': 2807 indent += 1 2808 elif ch == '\t': 2809 indent += tabsize - (indent % tabsize) 2810 elif ch in '\r\n': 2811 continue # skip all-whitespace lines 2812 else: 2813 break 2814 else: 2815 continue # skip all-whitespace lines 2816 if DEBUG: print("dedent: indent=%d: %r" % (indent, line)) 2817 if margin is None: 2818 margin = indent 2819 else: 2820 margin = min(margin, indent) 2821 if DEBUG: print("dedent: margin=%r" % margin) 2822 2823 if margin is not None and margin > 0: 2824 for i, line in enumerate(lines): 2825 if i == 0 and skip_first_line: continue 2826 removed = 0 2827 for j, ch in enumerate(line): 2828 if ch == ' ': 2829 removed += 1 2830 elif ch == '\t': 2831 removed += tabsize - (removed % tabsize) 2832 elif ch in '\r\n': 2833 if DEBUG: print("dedent: %r: EOL -> strip up to EOL" % line) 2834 lines[i] = lines[i][j:] 2835 break 2836 else: 2837 raise ValueError("unexpected non-whitespace char %r in " 2838 "line %r while removing %d-space margin" 2839 % (ch, line, margin)) 2840 if DEBUG: 2841 print("dedent: %r: %r -> removed %d/%d"\ 2842 % (line, ch, removed, margin)) 2843 if removed == margin: 2844 lines[i] = lines[i][j+1:] 2845 break 2846 elif removed > margin: 2847 lines[i] = ' '*(removed-margin) + lines[i][j+1:] 2848 break 2849 else: 2850 if removed: 2851 lines[i] = lines[i][removed:] 2852 return lines 2853 2854 2855def _dedent(text, tabsize=8, skip_first_line=False): 2856 """_dedent(text, tabsize=8, skip_first_line=False) -> dedented text 2857 2858 "text" is the text to dedent. 2859 "tabsize" is the tab width to use for indent width calculations. 2860 "skip_first_line" is a boolean indicating if the first line should 2861 be skipped for calculating the indent width and for dedenting. 2862 This is sometimes useful for docstrings and similar. 2863 2864 textwrap.dedent(s), but don't expand tabs to spaces 2865 """ 2866 lines = text.splitlines(1) 2867 _dedentlines(lines, tabsize=tabsize, skip_first_line=skip_first_line) 2868 return ''.join(lines) 2869 2870 2871class _memoized(object): 2872 """Decorator that caches a function's return value each time it is called. 2873 If called later with the same arguments, the cached value is returned, and 2874 not re-evaluated. 2875 2876 http://wiki.python.org/moin/PythonDecoratorLibrary 2877 """ 2878 def __init__(self, func): 2879 self.func = func 2880 self.cache = {} 2881 2882 def __call__(self, *args): 2883 try: 2884 return self.cache[args] 2885 except KeyError: 2886 self.cache[args] = value = self.func(*args) 2887 return value 2888 except TypeError: 2889 # uncachable -- for instance, passing a list as an argument. 2890 # Better to not cache than to blow up entirely. 2891 return self.func(*args) 2892 2893 def __repr__(self): 2894 """Return the function's docstring.""" 2895 return self.func.__doc__ 2896 2897 2898def _xml_oneliner_re_from_tab_width(tab_width): 2899 """Standalone XML processing instruction regex.""" 2900 return re.compile(r""" 2901 (?: 2902 (?<=\n\n) # Starting after a blank line 2903 | # or 2904 \A\n? # the beginning of the doc 2905 ) 2906 ( # save in $1 2907 [ ]{0,%d} 2908 (?: 2909 <\?\w+\b\s+.*?\?> # XML processing instruction 2910 | 2911 <\w+:\w+\b\s+.*?/> # namespaced single tag 2912 ) 2913 [ \t]* 2914 (?=\n{2,}|\Z) # followed by a blank line or end of document 2915 ) 2916 """ % (tab_width - 1), re.X) 2917_xml_oneliner_re_from_tab_width = _memoized(_xml_oneliner_re_from_tab_width) 2918 2919 2920def _hr_tag_re_from_tab_width(tab_width): 2921 return re.compile(r""" 2922 (?: 2923 (?<=\n\n) # Starting after a blank line 2924 | # or 2925 \A\n? # the beginning of the doc 2926 ) 2927 ( # save in \1 2928 [ ]{0,%d} 2929 <(hr) # start tag = \2 2930 \b # word break 2931 ([^<>])*? # 2932 /?> # the matching end tag 2933 [ \t]* 2934 (?=\n{2,}|\Z) # followed by a blank line or end of document 2935 ) 2936 """ % (tab_width - 1), re.X) 2937_hr_tag_re_from_tab_width = _memoized(_hr_tag_re_from_tab_width) 2938 2939 2940def _xml_escape_attr(attr, skip_single_quote=True): 2941 """Escape the given string for use in an HTML/XML tag attribute. 2942 2943 By default this doesn't bother with escaping `'` to `'`, presuming that 2944 the tag attribute is surrounded by double quotes. 2945 """ 2946 escaped = _AMPERSAND_RE.sub('&', attr) 2947 2948 escaped = (attr 2949 .replace('"', '"') 2950 .replace('<', '<') 2951 .replace('>', '>')) 2952 if not skip_single_quote: 2953 escaped = escaped.replace("'", "'") 2954 return escaped 2955 2956 2957def _xml_encode_email_char_at_random(ch): 2958 r = random() 2959 # Roughly 10% raw, 45% hex, 45% dec. 2960 # '@' *must* be encoded. I [John Gruber] insist. 2961 # Issue 26: '_' must be encoded. 2962 if r > 0.9 and ch not in "@_": 2963 return ch 2964 elif r < 0.45: 2965 # The [1:] is to drop leading '0': 0x63 -> x63 2966 return '&#%s;' % hex(ord(ch))[1:] 2967 else: 2968 return '&#%s;' % ord(ch) 2969 2970 2971def _html_escape_url(attr, safe_mode=False): 2972 """Replace special characters that are potentially malicious in url string.""" 2973 escaped = (attr 2974 .replace('"', '"') 2975 .replace('<', '<') 2976 .replace('>', '>')) 2977 if safe_mode: 2978 escaped = escaped.replace('+', ' ') 2979 escaped = escaped.replace("'", "'") 2980 return escaped 2981 2982 2983# ---- mainline 2984 2985class _NoReflowFormatter(argparse.RawDescriptionHelpFormatter): 2986 """An argparse formatter that does NOT reflow the description.""" 2987 def format_description(self, description): 2988 return description or "" 2989 2990 2991def _test(): 2992 import doctest 2993 doctest.testmod() 2994 2995 2996def main(argv=None): 2997 if argv is None: 2998 argv = sys.argv 2999 if not logging.root.handlers: 3000 logging.basicConfig() 3001 3002 parser = argparse.ArgumentParser( 3003 prog="markdown2", description=cmdln_desc, usage='%(prog)s [PATHS...]', 3004 formatter_class=_NoReflowFormatter 3005 ) 3006 parser.add_argument('--version', action='version', 3007 version='%(prog)s {version}'.format(version=__version__)) 3008 parser.add_argument('paths', nargs='*', 3009 help=( 3010 'optional list of files to convert.' 3011 'If none are given, stdin will be used' 3012 )) 3013 parser.add_argument("-v", "--verbose", dest="log_level", 3014 action="store_const", const=logging.DEBUG, 3015 help="more verbose output") 3016 parser.add_argument("--encoding", 3017 help="specify encoding of text content") 3018 parser.add_argument("--html4tags", action="store_true", default=False, 3019 help="use HTML 4 style for empty element tags") 3020 parser.add_argument("-s", "--safe", metavar="MODE", dest="safe_mode", 3021 help="sanitize literal HTML: 'escape' escapes " 3022 "HTML meta chars, 'replace' replaces with an " 3023 "[HTML_REMOVED] note") 3024 parser.add_argument("-x", "--extras", action="append", 3025 help="Turn on specific extra features (not part of " 3026 "the core Markdown spec). See above.") 3027 parser.add_argument("--use-file-vars", 3028 help="Look for and use Emacs-style 'markdown-extras' " 3029 "file var to turn on extras. See " 3030 "<https://github.com/trentm/python-markdown2/wiki/Extras>") 3031 parser.add_argument("--link-patterns-file", 3032 help="path to a link pattern file") 3033 parser.add_argument("--self-test", action="store_true", 3034 help="run internal self-tests (some doctests)") 3035 parser.add_argument("--compare", action="store_true", 3036 help="run against Markdown.pl as well (for testing)") 3037 parser.set_defaults(log_level=logging.INFO, compare=False, 3038 encoding="utf-8", safe_mode=None, use_file_vars=False) 3039 opts = parser.parse_args() 3040 paths = opts.paths 3041 log.setLevel(opts.log_level) 3042 3043 if opts.self_test: 3044 return _test() 3045 3046 if opts.extras: 3047 extras = {} 3048 for s in opts.extras: 3049 splitter = re.compile("[,;: ]+") 3050 for e in splitter.split(s): 3051 if '=' in e: 3052 ename, earg = e.split('=', 1) 3053 try: 3054 earg = int(earg) 3055 except ValueError: 3056 pass 3057 else: 3058 ename, earg = e, None 3059 extras[ename] = earg 3060 else: 3061 extras = None 3062 3063 if opts.link_patterns_file: 3064 link_patterns = [] 3065 f = open(opts.link_patterns_file) 3066 try: 3067 for i, line in enumerate(f.readlines()): 3068 if not line.strip(): continue 3069 if line.lstrip().startswith("#"): continue 3070 try: 3071 pat, href = line.rstrip().rsplit(None, 1) 3072 except ValueError: 3073 raise MarkdownError("%s:%d: invalid link pattern line: %r" 3074 % (opts.link_patterns_file, i+1, line)) 3075 link_patterns.append( 3076 (_regex_from_encoded_pattern(pat), href)) 3077 finally: 3078 f.close() 3079 else: 3080 link_patterns = None 3081 3082 from os.path import abspath, dirname, exists, join 3083 markdown_pl = join(dirname(dirname(abspath(__file__))), "test", 3084 "Markdown.pl") 3085 if not paths: 3086 paths = ['-'] 3087 for path in paths: 3088 if path == '-': 3089 text = sys.stdin.read() 3090 else: 3091 fp = codecs.open(path, 'r', opts.encoding) 3092 text = fp.read() 3093 fp.close() 3094 if opts.compare: 3095 from subprocess import PIPE, Popen 3096 print("==== Markdown.pl ====") 3097 p = Popen('perl %s' % markdown_pl, shell=True, stdin=PIPE, stdout=PIPE, close_fds=True) 3098 p.stdin.write(text.encode('utf-8')) 3099 p.stdin.close() 3100 perl_html = p.stdout.read().decode('utf-8') 3101 sys.stdout.write(perl_html) 3102 print("==== markdown2.py ====") 3103 html = markdown(text, 3104 html4tags=opts.html4tags, 3105 safe_mode=opts.safe_mode, 3106 extras=extras, link_patterns=link_patterns, 3107 use_file_vars=opts.use_file_vars, 3108 cli=True) 3109 sys.stdout.write(html) 3110 if extras and "toc" in extras: 3111 log.debug("toc_html: " + 3112 str(html.toc_html.encode(sys.stdout.encoding or "utf-8", 'xmlcharrefreplace'))) 3113 if opts.compare: 3114 test_dir = join(dirname(dirname(abspath(__file__))), "test") 3115 if exists(join(test_dir, "test_markdown2.py")): 3116 sys.path.insert(0, test_dir) 3117 from test_markdown2 import norm_html_from_html 3118 norm_html = norm_html_from_html(html) 3119 norm_perl_html = norm_html_from_html(perl_html) 3120 else: 3121 norm_html = html 3122 norm_perl_html = perl_html 3123 print("==== match? %r ====" % (norm_perl_html == norm_html)) 3124 3125 3126if __name__ == "__main__": 3127 sys.exit(main(sys.argv))