“Narou.rb” is a Web novel management tool that I personally love using. Running inside a Docker container, it’s a reliable partner that automatically collects daily updates and converts them into e-books. However, the other day, when I tried to download a specific new novel, it was stubbornly rejected.<\/p>\n
The logs mercilessly displayed the following:<\/p>\n
[ERROR] \u5c0f\u8aac\u304c\u524a\u9664\u3055\u308c\u3066\u3044\u308b\u304b\u975e\u516c\u958b\u306a\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059\n[ERROR] https:\/\/ncode.syosetu.com\/n8281jr\/ \u306e\u76ee\u6b21\u30c7\u30fc\u30bf\u304c\u53d6\u5f97\u51fa\u6765\u307e\u305b\u3093\u3067\u3057\u305f<\/pre>\nIt would be easy to give up and think, “Ah, it must have been deleted.” However, when accessing it via a browser, that novel was actively serialized and doing well. This was a challenge letter from the server. This record details how I tracked down the cause of Narou.rb’s false positive\u2014thinking it was “deleted”\u2014and resolved it by modifying the configuration file (YAML).<\/p>\n
Mission: Clear the False Accusation and Resume Collection<\/h2>\n
The objective this time is to “correctly recognize an existing novel and make the download succeed.”<\/p>\n
In web scraping tools, acquisition failures due to target site design changes or fluctuations in HTML structure are inevitable. However, since the error message stated it was “deleted,” the tool judged that “the page itself could not be found (or could not be parsed).”<\/p>\n
The goal (quantitative target) is as follows:<\/h3>\n
\n
- Successful download of the target novel (ID: n8281jr) (100% progress)<\/li>\n
- Robust configuration that won’t throw errors even for novels with similar HTML structures in the future<\/li>\n<\/ol>\n
Investigation Begins: Why Is It “Invisible”?<\/h2>\n
First, I checked the connection from the server (Docker container) to the internet. Using the curl command, I tried accessing Narou’s server directly from inside the container.<\/p>\n
docker-compose exec narou curl -I https:\/\/ncode.syosetu.com\/n8281jr\/<\/pre>\nThe result was HTTP\/1.1 200 OK. There were no problems with the communication path, and it was not blocked by a WAF (firewall) like Cloudflare. In other words, the HTML data had arrived safely.<\/p>\n
The fact that data arrived yet could not be “acquired” means that the “map for page analysis (regular expressions)” held by Narou.rb and the actual “terrain (HTML)” were mismatched.<\/p>\n
Suspects Under Investigation<\/h2>\n
At first, I suspected changes in the format of the table of contents list or update dates. Were there line breaks next to the dates? Did the tag class names change? I carefully reviewed subtitles (table of contents definition) and nu (update date definition) in the configuration file (ncode.syosetu.com.yaml), but lacked a smoking gun.<\/p>\n
However, light shines nearest the candlestick. The parsing failed at the most fundamental part: the “title.”<\/p>\n
Solution: The Pitfall Called Line Breaks<\/h3>\n
In Narou.rb’s configuration file (YAML), the rule for fetching titles was written as follows:<\/p>\n
Before Modification (Problematic Description)<\/h3>\n
# \u30bf\u30a4\u30c8\u30eb\nt: <h1 class="p-infotop-title"><a href=".+?">(?<title>.+?)<\/a><\/h1><\/pre>\nAs a regular expression, this expects that “everything from the h1 tag to the closing a tag is connected in a single line.” This is because the dot (.) in regular expressions normally only matches “any character except a line break.”<\/p>\n
However, when I checked the HTML source of the novel that failed to download this time, it looked like this:<\/p>\n
Actual HTML<\/h3>\n
<h1 class="p-infotop-title">\n<a href="...">\n\u5d29\u58ca\u4e16\u754c\u306e\u9b54\u6cd5\u6756\u8077\u4eba\n<\/a>\n<\/h1><\/pre>\nThere are line breaks between the h1 and a tags, and before and after the title. Because of this “single line break,” the regular expression above (.+?) failed to match, leading Narou.rb to the false conclusion that “title not found = this is not a novel page = it must have been deleted.”<\/p>\n
Applying the Patch<\/h3>\n
Once the cause is known, fixing it is easy. We just need to make the YAML description support “multiple lines (line breaks).” Using the volumes feature of docker-compose, we mount the modified YAML file into the container.<\/p>\n
After Modification (Fixed YAML)<\/h3>\n
# \u30bf\u30a4\u30c8\u30eb\n# |- \u3092\u4f7f\u3046\u3053\u3068\u3067\u3001\u6539\u884c\u3092\u542b\u3093\u3060\u6587\u5b57\u5217\u3068\u3057\u3066\u5b9a\u7fa9\u3059\u308b\nt: |-\n<h1 class="p-infotop-title">\n<a href=".+?">(?<title>.+?)<\/a>\n<\/h1><\/pre>\nBy inserting line breaks to match the HTML structure (or using a regular expression containing line breaks like [\\s\\S]+?), the parser can now correctly recognize the title.<\/p>\n
Applying this modified file, I executed the download command again.<\/p>\n
docker-compose exec narou narou download n8281jr<\/pre>\n<\/p>\nAs a result, as if to belie its previous stubborn rejections, the download progress bar began to run.<\/p>\n
Summary from Nando Kobo<\/h2>\n
This trouble was not a system bug, but a collision between “overly strict definitions” and “HTML fluctuations.”<\/p>\n
\n
- Doubt error logs: Even if told “it has been deleted,” first verify its existence using a browser and curl.<\/li>\n
- The trap of regular expressions: The . (dot) is not almighty. In HTML parsing, “line breaks” can always become an enemy.<\/li>\n
- YAML flexibility: By slightly rewriting the definition file, the tool springs back to life.<\/li>\n<\/ol>\n
Even on a narrow server like a walk-in closet (Nando), a comfortable reading environment is maintained by accumulating such small “repairs.” If you ever encounter a phenomenon where “only a certain novel won’t download,” please definitely suspect the “regular expression around the title” in the definition file.<\/p>\n
Wishing you a good server life.<\/p>\n","protected":false},"excerpt":{"rendered":"
“Narou.rb” is a Web novel management tool that I personally love using. Running inside a Docker co […]<\/p>\n","protected":false},"author":4,"featured_media":2582,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_locale":"en_US","_original_post":"https:\/\/donguri3.net\/?p=2580","footnotes":""},"categories":[1171],"tags":[118,18,19,52,291,10,381,397,1042],"class_list":["post-5315","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-virtualization-container-wsl","tag-cloudflare","tag-docker","tag-docker-compose","tag-https","tag-yaml","tag-server","tag-381","tag-397","tag-1042","en-US"],"_links":{"self":[{"href":"https:\/\/donguri3.net\/wp-json\/wp\/v2\/posts\/5315","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/donguri3.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/donguri3.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/donguri3.net\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/donguri3.net\/wp-json\/wp\/v2\/comments?post=5315"}],"version-history":[{"count":1,"href":"https:\/\/donguri3.net\/wp-json\/wp\/v2\/posts\/5315\/revisions"}],"predecessor-version":[{"id":5318,"href":"https:\/\/donguri3.net\/wp-json\/wp\/v2\/posts\/5315\/revisions\/5318"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/donguri3.net\/wp-json\/wp\/v2\/media\/2582"}],"wp:attachment":[{"href":"https:\/\/donguri3.net\/wp-json\/wp\/v2\/media?parent=5315"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/donguri3.net\/wp-json\/wp\/v2\/categories?post=5315"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/donguri3.net\/wp-json\/wp\/v2\/tags?post=5315"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}