TrickJarrett.com

HTML to Markdown

We're officially a Markdown blog! For those unfamiliar with Markdown, it's a 'markup language.' That is, like HTML, it defines how a document will look - using a very simple system. I'll dive into why I'm doing this later in the post. For you all, you should see (almost) no change on the blog, most everything I discuss here is relating to the backend management of it.

A few days ago, I quickly worked out the code which would take every entry in the blog and convert them from HTML to Markdown. I also took the opportunity to fix some content errors in the site; though it's not all fixed. Before running the conversion, I did a lot of testing to be confident it wouldn't screw up the database - but, having done that, I still took an export of the database as backup before running it. You can never be too careful when dealing with something that could potentially ruin your website.

This morning, after the entries in the database were converted, I switched my admin interface to a different editor which supports Markdown. I am currently going with Editor.md which looks to be a markdown editor made for the Chinese market. It's quite robust, though I have a few things I still need to figure out. Such as, when writing this post, I went to use a modal and the UI popped up in Chinese! Whoops. Thankfully it was a quick fix as they provide a bit of code which fully translates the UI into English.

Once the editor was in place, and I had confirmed all of my normal admin functionality worked, I then finished implementing the publishing which would turn the Markdown into HTML. Thankfully the Parsedown library for PHP is robust and simple. It only took a few lines of code to implement it.

That was it. Overall, very straight forward, and now my blog is future proofed. The future-proofing is one of the major upsides of using Markdown.

The editor I was using before was called Quill, and while it's a very powerful rich-text editor,the HTML it created was not perfect. This isn't a problem limited to Quill, many rich text editors struggle with this. I still remember the nightmare code that Microsoft Word would produce if you saved your document as an HTML file. So, with Quill, I was having to do code cleanups and figure out workarounds for some of its quirks.

Ultimately, this is not a huge thing. Many other blogs store their posts in HTML. I just like knowing it is in Markdown for flexibility. And, making Markdown the core structure of the backend enables me to potentially do some other things for inputting new entries. Dave Winer, the father of RSS, has a neat thing he put together where he can tweet a thread in markdown and his blog will pick it up and import it as an entry. I don't currently plan to do that with this blog, but it's an example of where making Markdown the editorial core enables some greater flexibility.

Glowbug coding morning

I have, for the past 3 days, been banging my head against the wall.

I utilize an app called 'Wallabag' which is a self-hosted version of Pocket, for storing articles for later reading.

The issue I'm running into is that I want to hook a bit of code I'm writing up to my Wallabag. Wallabag has a built-in API setup. It's documented and works for a number of things already. I'm attempting to create my own. But, for some unknown reason, the path /oauth/v2/token, which works on their hosted setup and for other people who are self hosting, is not working for me.

I've been working through possible reasons. I even contacted my webhost to see if there was possibly some sort of intra-server limitation that would stop it. But, no luck.

My current theory is that there is something messed up with my install. I installed it via 3rd-party tool called Scriptaculous. So, in attempting to fix that, I go about installing it via their prescribed way of doing it. And, because this problem has become a huge constant series of obstructions, I end up with errors as part of install. Fun!

When I ran into the wall on it, I turned to some other bits of coding. Namely, getting me closer to the transition to Markdown rather than HTML on the backend. I have just finished the script which will convert all the posts from HTML to markdown, as well as fixing some other issues.

Tonight I'll finish adding the Markdown parsing to publication and then change the admin entry editor from Quill to a Markdown one. Once that's done, we'll run the conversion.

It occurred to me, as I laid in bed last night, that I can recreate one of the most useful integrations I ever had with IFTTT. I used to have IFTTT grab any Tweet which I liked (as in, clicked the heart / star on), and then they would grab the URL from that tweet and save it to my Pocket for later reading.

IFTTT is not what it used to be, and I'm now using my self-hosted Wallabag, and also I have more coding prowess and power - so I this morning I spent a bit and have my own implementation of this nearly complete. I'm running into some issues with connecting to the Wallabag API, but once I have that sorted - it should be a very quick next few steps to get this up and running. We'll see.

I cannot stress enough that the key to a content consumption workflow like what I've described in the path relies on making the action of capturing content to read later as seamless and smooth as possible.

Fascinating thread about the business of booking tables at restaurants

Upcoming Glowbug Work

Now that I hit the main milestones for Pick'em, I have spent much of the past week or so working on stuff for Glowbug. It's really a tool for me. I've talked about the work I've already done. I'm listing these in no particular order:

Add ability to set custom URL path rather than use the hash - Some entries on this blog get their own page, which have a direct link to them. The vast majority of posts here don't and are only accessible through the day archives. But, up to now, if a page gets its own URL, it's a generated hash. When I started this blog engine, I didn't care about the URLs and just wanted it to be quick and easy. I've reached the point now where I'm ready to be able to add a bit more curation and want to be able to manually define a page's URL path.

Add Chapters for posts - This is an old old idea I've had for blogs. Especially blogs which are life journals. Having posts organized into chronological groupings automatically. More than just a day or week archive. But you would basically say "Chapter 1" or "Seattle" or whatever title makes sense at the time you start the chapter. And then posts would be added to it cumulatively until something occurs in your life and you turn to a new chapter. So, I want to finally implement that idea here for the life update and current event posts.

Combine multi line quotes - This is a bit of a smaller bug, but right now, if I input a multiparagraph quote, Glowbug renders each paragraph as their own quote block. Annoying. Also, not huge, and I might put it off as I plan on doing a larger rewrite of the post creation system. Moving away from the Rich HTML structure and reverting to a more simple Markdown based content entry basis. We'll see.

Add pages for tags - When I first added tagging to the blog, my intent was to go back and add logic for creating pages based on tags. I still want to do this. Though, I'll need to figure out how to handle the tags with a bunch of posts. I don't particularly want to paginate, but I might have to.

Make publishing smart by identifying all the pages which need to be updated - Another thing I've been putting off. Right now, when I post a single link, the backend rebuilds the entire site. That's silly and wasteful. I want to go through and figure out how to best identify the pages which are affected by this post and update them.

Fix Suggested Tags - The suggested tag functionality I currently have gets the job done, but it's crappy. I can do better. There are some bugs with it and I just need to sit down and take on fixing it.

Yesterday I polled my Twitter followers about whether the automated tweets updating them about recent posts:

[{embed}]https://twitter.com/trickjarrett/status/1528166144800018432\[{/embed}]

Overall the feedback was that they were fine. Which is better than a resounding amount of people telling me they were annoying. I did get a good suggestion to add a tag which people could filter out, so, I did that. Along with that, I improved the overall coding.

First, I fixed it so the grammar of the tweets will be correct regardless of the number of posts made, or tags used. Previously, on days with few posts or tags, would lead to some poorly worded tweets.

Second, I refined the system for how it chooses tags to highlight. It's still far from perfect, but it first looks at the tags used in the day and ranks them by frequency of use. If there is hierarchy there, it goes with it. If not (aka, the tags are all used once) then it falls back to ranking tags by their frequency of use across the entire blog. Still not perfect, but better than the purely random highlight of tags.

Lastly, I cleaned up and refactored the code. In fact, as I was writing this blog post, I had to stop and go back as I realized I could further improve the refactoring of the code.

Automated summaries and keyword identification from articles

Well, this bit of coding has been a journey.

This week I've been working on code which automates a pipeline from my Wallabag (a self-hosted web app that enables later-reading of articles online, like Mozilla's Pocket app) and my blog. While working on this, I discovered this GitHub project for 'TextRank' which takes a body of text and it will attempt to summarize the text, as well as identify keywords in the text. It is definitely not perfect, but it is useful for a first iteration of the concept.

I've been trying to integrate it into my code over the past few days to infuriatingly little success. This afternoon, I finally was able to get it - but only after getting on StackOverflow to ask about what I was missing. As I was doing so, I realized I had asked a question about the exact same issue six years ago.

I am thrilled to have found my solution, and mortified that I had forgotten about this.

So, the code now does two things:

First, it generates a summary for the text. These summaries will not always be great, but the hope is that they are a net value-add for this automation of the system. My intent is that these summaries will only be present until I come back and revise the content for the posts, either determining the summary is not needed or I replace it with something I write. We'll see.

Second, it does keyword analysis. I then take the top keywords it identifies and also keywords which already exist as tags, and add them to the new post. Again, not a perfect system, but better than nothing, and something I can iterate on.

Interestingly, I spent a summer in college working with Dr. Lonnie Harvel, during which I contributed to a paper he published titled, "Using student-generated notes as an interface to a digital repository." At the time, Georgia Tech, had just rolled out lecture recording and automated transcripts of the video with time stamps, etc. We were working on stuff that would further improve that system.

My main contribution there was work on code that looked at the transcript and identified keywords. It's been so long, I don't remember the full details of what I came up with, but I do recall it being something relating to a number of different things, like frequency of word usage in the text, word length as well as number of syllables (my thinking was that the bigger words would tend to be more important.) Granted, the context there was in identifying words that would do well in being sign posts for lecture transcriptions which is slightly different than identifying the most relevant and salient keywords for taxonomy.

In any case, it is interesting to come back to something I had done some research on back in college. I'm looking forward to seeing how the new implementation works on the blog and we'll see about improving and refining it from here.

Edit (12:29am): It took less than four hours before I decided to rework the system. I recalled there was a bot on Reddit which would pop up and attempt to share summaries in reply to links to articles. I tracked it down and found that it made use of another site that does summaries, smmry.com. After investigating, I found I could have the API use it up to 100 times a day, which should be plenty for my purposes. Their tool for providing keywords is slightly too opaque for my uses, currently, though I might reconsider and use it in conjunction with the current tool - though I'm not convinced that will be overly helpful yet. We'll see.

I have had an idea for a new RSS reader. It's a tool I want, but I don't really want to build. Currently my RSS reader pulls in 97 different feeds from varying sources, mainstream news, regional news, blogs, link aggregators, etc. One of my frustrations is when I see multiple articles on the same topic sporadically through my feed. Granted, this isn't a huge issue, but each successive article is an increasing volume of noise in the feed.

It makes sense, because I am purposefully pulling in diverse feeds, multiple places are covering a topic. And if it is an important topic, then it is good for multiple perspectives, etc. This issue is one thing preventing me from actually pulling in more feeds, with this system, the posts on the same topic suddenly become a potential value add rather than a detractor.

But at the same time, I'd really like to smooth the feed and instead see a singular entry on the topic with a prominent inclusion of "these other posts are discussing it as well."

So, I've had this idea for a while. Last night I think I found an open source project which would get me much closer and be, at the least, a first tool in looking at posts and identifying ones which have overlap. I will have to explore it further this weekend.

ntfy.sh - Send push notifications to your phone via PUT/POST

This is an interesting service I just came across. I've been thinking about figuring out how to send notifications to my phone from some web apps I am working on, this enables that via a free service.

Glowbug + Wallabag

I just hacked in a new feature on the frontpage of this blog, I've integrated my Wallabag with this blog! Wallabag is a web application allowing you to save web pages for later reading, there is a popular tool called Pocket which does the same thing. The difference between Wallabag and Pocket is that Wallabag is something I can self-host.

The first level of this integration is that you can see what articles are currently sitting unread in my Wallabag. These are things that I come across through the day and I throw them in there to come back to at some point, maybe. I make no guarantees on the quality of them as they are often saved either based purely on the subject, or maybe by reading the first paragraph or so.

I have a few further ideas:

• I could use the 'Star' function in Wallabag to grab articles from my Wallabag automatically to be published here.
• A dedicated RSS feed for my Wallabag
• An automated newsletter for the links

Last night I pushed the second "big" feature for my Pick'em site. It went shockingly smoothly. I expected it to be mostly good but pretty much the only hiccup was an errant curly brace. I was shocked.

The code is the about 70% of the full feature and is mainly the functionality that is needed for the league to continue to function as planned. This weekend I'll work on the remaining 30%, which is largely administrative tools and integrating it further into a few other nooks and crannies on the site.

With the deployment though, I find myself deciding on my next coding project. I think I'll do a bit of work on 'Clerk' which is, right now, just my weight and health tracker. It needs a bit of love and I want to expand it's functionality some. But that isn't too large a project. I think after that I'm going to start work on what I'm calling 'Kontakto' which will be a personal CRM tool. It will be my second foray into making an Android app.

Project Statuses

Taking a cue from Brandon Sanderson I thought I should start doing reports on the status of my projects. He has status bars which just get updated on his site, I'm not doing that (yet?) - I'm doing a post which summarizes the status of various projects underway. My intent is for this to be a monthly check-in.

Coding Projects

Pick'em

Status: Active

Summary: The site is up. There are two major features which I need to work on, one of which is underway. They are: Redoing the Match page & Implementing split-season scoring. The former involves rebuilding the page where users view and pick matches, as well as implementing the pick logic client side so they don't have to try and wait for the server to reject their picks.

Next Steps & Timeline: Finish my work on rebuilding the match page (2 weeks?) and then work on revamping player scoring to implement split-season scoring (finish by end of April)

Glowbug

Status: Active, mostly idle

Summary: My homebrew blogging engine. The work this week was the most I've done in a long time. The system works, it's just about making it better at this point.

Next Steps & Timeline: Nothing major, just coding when it strikes me. No timelines on it.

Clerk

Status: Mostly idle

Summary: Currently my personal weight and exercise tracker, but long term it's meant to be my full personal hub for managing daily life. It's up and I use it daily (need to weigh in today, in fact.) There are some active bugs which need squashing, but they are low priority.

Next Steps & Timeline: Bugs will be squashed when I need a coding distraction from whatever I'm working on.

Fixture Picker

Status: Needs rewrite

Summary: My fixture picker works but it isn't great and it needs a rewrite. This project lost a lot of stream as I thought it might have business viability but some surveys and research led me to abandon it as not overly viable.

Next Steps & Timeline: Unclear. I might abandon this project altogether simply to clear the deck for other things.

Cement

Status: Idea phase

Summary: A homebrew collection management tool. This promises to be a coding project near the size of Pick'em if not larger. I have done a lot of thinking and have a database schema but I haven't begun on this project. I'm earmarking it as a project for the 2nd half of 2022.

Next Steps & Timeline: Want to start work before 2023 and have this be my primary project next year.

Secret Project

Status: Prototyping

Summary: There's a fairly new project I've started toying with in the past few weeks. As I think it might have commercial viability, I'm keeping wraps on it and have only discussed it with a few people.

Next Steps & Timeline: Unclear, doing research and planning.

Writing Projects

Urban Fantasy 1, "Thorn" - My NaNoWriMo book for 2020. It's "complete" and just needs editing and tightening of the plot. Thorn is the working title I started with, I'm not settled on it as the final title.

Urban Fantasy 2, "Untitled" - I started it for NaNoWriMo 2021 but have not finished it. I hit a bit of a wall and am struggling to find the next section of the book.

Sci-Fi 1, "The Stars in the Canopy" - I've had the idea for a science fiction story but have not really started on it yet.

Integrating COUNT() into a nested MySQL query

I'm working on Glowbug's admin side. One of the features I've wanted to add is for it to recommend existing tags which are similar to ones I've already entered into a post. I'm working on the query which will pull this info up and decided to turn to StackOverflow to see if there is a better way to do it than I am currently. We'll see what the responses say.

Update 9:15am - StackOverflow comes through and gives me the answer.

Coding Updates

This will be a bit of a meandering post, not in the sense that I don't know where I'm going with it, but that I plan to hit a few tangentially related topics along the way. Obviously my last post was about how I had revamped my project boards for my main coding projects, so coding is on my mind.

I have found Minecraft scratches a lot of the itch I serve by coding on my own projects. Coding projects, to me, are my equivalent of the people who sew, or do woodworking, or work on cars, etc. It's a hobby and craft that I find satisfaction in the work and the end product. And Minecraft's world building fills much the same mind space for me. It's gotten to the point where I have to force myself to put Minecraft aside and turn back to these side projects lest they fall by the wayside and into disrepair.

It's also notable that the only one of my code projects which is used by other people is the main motivator for this. I run a self-hosted custom MLS Pick'em league for friends and acquaintances. With the next MLS season on the horizon, I have to get moving on updates and fixes in preparation for the next season.

And so, after several months of going hardcore on Minecraft, I have to move it to a back seat for a bit and instead return to my hobby of coding in the hours after work.

My current plotted course for this work is:

1. MLS Pick'em - It will be my exclusive (well, probably) coding project until the new season kicks off. Then I'll let it fall back in priority to maintenance rather than working for new features. Putting arbitrary dates on it, I'm expecting to focus on it until middle of March and then it gets backburnered.
2. Clerk - My personal tracker, it's a small project and I have a few updates and new features I'd like to add. This stuff should honestly take a week or two, so probably through the rest of March.
3. Glowbug - This blog hasn't had any serious work from me for nearly a year now. And I do have work I want to do on it. This ranges from front facing layout and design, adding visitor search, importing tweets into daily archives so they get archived here, and other sundry items. Assuming the time estimates for Pick'em and Clerk are correct (and we know how good coders are at estimating work time) I think the work I want to do here would probably be all of April and could extend into May depending on how ambitious I want to be.
4. CEMENT (CollEction ManagEr aNd Tracker) - I wrote about a flexible collection manager tool last year, and it is still an idea I have been turning over now that my Magic collection is sorted. So, if I'm still actively thinking about it, then it is time for me to probably act on it. It would have utility for my Magic collection, Katie's Pez collection, our boardgames, and various other collections we have. In scale, I think it would end up being closer to Pick'em as far as size, there's a lot of front end which would need to be done for it. We'll see. For Cement, I can easily see it being a 3 month project to get it up and running and then it becomes about maintenance and adding new features.

Assuming my time estimates are correct, and that life cooperates (meaning there is nearly a zero percent chance I'm right) I have coding plans up through July. We'll see.

Revamping my Code Project Tracking

A few years ago, I set up simple project boards in my Notion. This allowed me to track issues with my various coding projects, and make reminders for improvements or new feature ideas.

This morning, after waking up abnormally early, I sat down and revamped how it works. Previously, they were a number of individual tables. It worked fine, but the biggest issue was when I created a new project I had to go and literally manually recreate the structure for each new project to track its own tickets. Again, not a huge issue since I add scant few projects. But, it annoyed me.

So, today, I redid the system to contain it all into a single larger table that I can then filter, or group, by project. I am also hoping this will allow me to do some smarter filtering and prioritization for coding.

New Project Idea - Scrib

I make frequent use of Google's Keep as my place for short easy notes. I also sometimes use Samsung Notes, as a by product of my writing on my phone (Note 9), but those are almost entirely scribbles as I lay in bed and want to write an idea down.

I want a note taking tool which lets me grab random thoughts and save them. They might be quotes, or ideas, or reminders. They might often be half-baked.

This idea actually came as I am rethinking my personal organization and figuring out how best to build it.

Goals

1. Be able to scribble a note within two clicks on my phone.
2. Make the referencing of these notes more useful

#2 is where both Keep and Samsung's notes tool falls down. They are just the digital equivalent of yellow sticky notes. I want to have it be smarter at organizing notes by date, as well as by identifying keywords. If the word 'D&D' appears in multiple notes, then I want that word linked to a page which pulls all notes which use that word. I'd like to do it even for phrases, but am not sure how taxing that is processing wise.

I'm going to keep doodling on a design for the project, and then I'll add it to my coding roadmap.

Years ago, I coded a very simple PHP tool for the easy making of a Secret Santa list. I put in people and email addresses, and it would pick who got who and send it to everyone.

Today, while doing something completely different and wholly unrelated to Christmas, coding, or anything. I suddenly realized the way I did it was overly complicated and dumb. Sigh.

I basically had it to random matching on each person of the front of the array, looping until it had randomly picked everyone in the array as givers and receivers.

I just realized that the easiest way for me to do this is to put everyone into an array, shuffle the array, and then just have the first person give to the second person, the second person give to the third person, the third to the fourth, and then the last person gives to the first.

Much simpler, cleaner, and faster.

Thinking about November for this year. Last year I wrote a (bad) novel. I have another idea for a novel that I might tackle this year. However I'm also toying with the idea of, rather than writing a book, binge-ing and writing an app for the phone. A month-long "hackathon" sort of thing.

The thing I'm considering is a podcast app. I have one "killer feature" for it, which is the ability to group/tag podcasts and then select playlists based on the group or tag. Want to listen to the news on your morning commute? Here's today's news podcasts from NPR, the New York Times, etc. Want something to laugh at while you work, hit your comedy group. Etc.

We'll see. I'm toying with the idea. I haven't written an Android app in years. Maybe I'll dive into it again for NaAppHackMo - National App Hackathon Month

Learning React.js

I have decided to improve my coding skills and catch up with some modern trends and tackle one of the most popular javascript frameworks, React. Up to now my front-end coding has been HTML + Jquery which ages me by decades compared to the modern day.

It's been a long time since I learned a new programming langue. I messed around a bit with Python recently but, to be honest, it felt very meh to me. Nothing revolutionary. To be clear, I recognize it as a great language and am not trashing it but I found no real hook for me.

So, in my effort to learn and adopt react as my front-end coding of choice, I have this rough development process in mind:

1. Tutorial project - Done! This was a very simple task manager tool that showed the basics.
2. A D&D HUD - This would be a tool I could use for managing combat in D&D as well as something to provide a nice front-end view for players to know who's up in the initiative, and out of combat, an easy way for me to show then stuff and convey bits of info about the world. We'll see how complicated it gets.
3. Fixture Picker - This was the project that actually caused me to jump into learning React. The way I want to take the tool really leans heavy on Javascript to allow people to personalize stuff. So I'll be using React as part of a full rewrite, making it the V4 of the Fixture Picker.
4. Glowbug - While I'm not blogging as much as I want to, I think this would be an excellent project to learn React and rebuild my admin functionality in it. The frontend will likely not change for readers, we'll see.
5. Pick'em - This is a further off project. I just did the major rewrite of my Pick'em site for this past season. I don't intend to do this in the short term, but it is a very likely development in the future. Probably 12-18 months down the line.

I am sure other projects will pop up, given time, but this is my road map of projects which I'll work on to develop my react.js chops. And of course, I'll keep this blog semi-updated with my progress.

Learning React.js

I have decided to improve my coding skills and catch up with some modern trends and tackle one of the most popular javascript frameworks, React. Up to now my front-end coding has been HTML + Jquery which ages me by decades compared to the modern day.

It's been a long time since I learned a new programming langue. I messed around a bit with Python recently but, to be honest, it felt very meh to me. Nothing revolutionary. To be clear, I recognize it as a great language and am not trashing it but I found no real hook for me.

So, in my effort to learn and adopt react as my front-end coding of choice, I have this rough development process in mind:

1. Tutorial project - Done! This was a very simple task manager tool that showed the basics.
2. A D&D HUD - This would be a tool I could use for managing combat in D&D as well as something to provide a nice front-end view for players to know who's up in the initiative, and out of combat, an easy way for me to show then stuff and convey bits of info about the world. We'll see how complicated it gets.
3. Fixture Picker - This was the project that actually caused me to jump into learning React. The way I want to take the tool really leans heavy on Javascript to allow people to personalize stuff. So I'll be using React as part of a full rewrite, making it the V4 of the Fixture Picker.
4. Glowbug - While I'm not blogging as much as I want to, I think this would be an excellent project to learn React and rebuild my admin functionality in it. The frontend will likely not change for readers, we'll see.
5. Pick'em - This is a further off project. I just did the major rewrite of my Pick'em site for this past season. I don't intend to do this in the short term, but it is a very likely development in the future. Probably 12-18 months down the line.

I am sure other projects will pop up, given time, but this is my road map of projects which I'll work on to develop my react.js chops. And of course, I'll keep this blog semi-updated with my progress.

Yesterday's Linux experiment did not last long. I ended up reverting the laptop to Lubuntu, I found that while Bunsenlabs looked nice, its support was lacking and I was not in the mood to dive deep into learning its intricacies.

That said, this morning I found myself diving deep on two arcane nerd tools: vim & lisp.

Vim is a code editor that is notorious for its power but also for its steep learning curve. It is a byproduct of the early days of unix and linux and it is a truly amazing piece of software, except that it is like learning a foreign language for all the jargon and key codes you have to learn. So, naturally, I am fascinated by it and eager to try and unlock it.

Secondly, Lisp, is a programming language. It is most widely used today for doing things like AI, but it is an old language. I had a course in college which used a version of Lisp. I have no real plans for a Lisp project, but I just find it interesting to explore things like this from time to time.

I have been using a FreshRSS install for a few months now, allowing myself to return to the realm of RSS feeds. And after those past few months I've dealt with a number of websites using truncated bodies of posts in the RSS feed, and occasionally clicking through to the full article on the website.

Well, last night I discovered a feature of FreshRSS that I had not known - and that is the ability to actually have it pull the full post body from the website through defining of the post body's CSS container. It's technically somewhat simple (I understand how to implement it, conceptually) but still having that function is amazing and has already made my RSS feed so much better.