path: root/served/words/atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

	<title>gennyble's writing</title>
	<subtitle>Technical writing; project updates; weeknotes</subtitle>
	<updated>2024-03-02T01:42:00-06:00</updated>

	<link rel="self" href="https://nyble.dev/words/atom.xml" type="application/atom+xml" />
	<id>https://nyble.dev/words/atom.xml</id>

	<author>
			<name>gennyble</name>
			<email>gen@nyble.dev</email>
	</author>

	
	<entry>
		<title>Akkoma Postgres Migration</title>
		<link href="https://nyble.dev/words/akkoma-postgres-migration.html" rel="alternate" type="text/html" />
		<id>https://nyble.dev/atom/writing-1/akkoma-postgres-migration</id>

		<published>2023-10-18T23:16:00-05:00</published>
		<updated>2023-10-18T23:16:00-05:00</updated>

		<content type="html">
&lt;p&gt;
&lt;i&gt;(i&apos;m going to say Pleroma a lot here where Akkoma might
	be correct for newly installed software, but my instance is
	a few years old and this is more of a telling-of-events than
	a guide)&lt;/i&gt;
&lt;/p&gt;
&lt;details class=&quot;tldr&quot;&gt;
	&lt;summary&gt;TL;DR; if you migrated your Akkoma&apos;s postgres and now you&apos;re getting timeouts&lt;/summary&gt;
	&lt;p&gt;
		It might need a reindex. Use &lt;code&gt;psql&lt;/code&gt; to connect
		to the database and run &lt;code&gt;REINDEX DATABASE akkoma;&lt;/code&gt;.
		This might take awhile.
	&lt;/p&gt;
&lt;/details&gt;
&lt;hr/&gt;
&lt;p&gt;
Recently I went about trying to get the services running on
my VPS to be happy in a gig of RAM. I did not achieve this,
but I found a solution that worked nearly as well.
&lt;/p&gt;
&lt;p&gt;
I wanted to try to scale my VPS, on the &quot;Linode 4GB&quot; plan, back down to a Nanode. It
started it&apos;s life as a Nanode but Akkoma - well, Pleroma then -
was greatly displeased with this and pegged my CPU at 100%. Since
my CPU usage lately peaks at 30% and averages 18%, this no longer
seems to be the case.
&lt;/p&gt;
&lt;p&gt;
To re-nanode, I had to fit in 1G of memory.
I managed to shave the 110M I needed
by asking &lt;code&gt;systemd-journald&lt;/code&gt; to stop using 80M of memory
&lt;i&gt;(it seemed to ignore my 10M plea, but it dropped by 30M so whatever)&lt;/i&gt;,
telling Postgres to max use 100M, and disabling things that
I as not actively using anymore.
&lt;/p&gt;
&lt;p&gt;
I didn&apos;t specifically want to learn the ins-and-outs of Postgres
performance tuning, so I used &lt;a href=&quot;https://pgtune.leopard.in.ua/&quot;&gt;pgtune&lt;/a&gt;
to give me the right config lines for 100M. It worked well!
&lt;/p&gt;
&lt;p&gt;
This was all for naught, though, because I couldn&apos;t get my
disk to fit under 25G, which was also a requirement of nanodeisation that I&apos;d
forgotten about. The database itself was 9.9G! You can
&lt;a href=&quot;https://docs.akkoma.dev/stable/administration/CLI_tasks/database/#prune-old-remote-posts-from-the-database&quot;&gt;Prune old remote posts&lt;/a&gt;
but I didn&apos;t really want to do that yet. It seems like the right
way to go, but I had one more trick.
&lt;/p&gt;
&lt;h2 id=&quot;two-of-them&quot;&gt;Two of Them?&lt;/h2&gt;
&lt;p&gt;
I have to keep a separate VPS around for another thing, and it gets
half a percent of CPU usage, which is... not a lot. All it does is serve
a single-page static site through Nginx. I could almost
certainly put this on the same server as all my things, but
I like having the separation.
&lt;/p&gt;
&lt;p&gt;
This does mean that I pay for almost an entire Nanode to do
very nearly nothing. 
&lt;/p&gt;
&lt;p&gt;
By putting Postgres on it I&apos;d lose the different-machine aspect
of the separation, but gain so much disk space and memory. The
single-page-static is still on a separate public IP which is
good enough for me!
&lt;/p&gt;
&lt;h3 id=&quot;setup-postgres&quot;&gt;Postgres Migration&lt;/h3&gt;
&lt;i&gt;(more of a recount of events than a guide, but written guidlike? just pay mind to the commands and you&apos;ll be fine)&lt;/i&gt;
&lt;p&gt;
Install Postgres on the new server. It doesn&apos;t have to be the
same major version since we&apos;re going to dump and restore the
database which is
&lt;a href=&quot;https://www.postgresql.org/docs/current/upgrading.html&quot;&gt;the recommended upgrade method anyway&lt;/a&gt;.
Don&apos;t forget to run &lt;code&gt;initdb&lt;/code&gt; and give your data
directory with the &lt;code&gt;-D&lt;/code&gt; flag. Run it under the
postgres user.
&lt;/p&gt;
&lt;p&gt;
Now create the database and role that you&apos;ll use. In my experience
these have to match the database you&apos;re migrating from. I followed
the &lt;a href=&quot;https://docs.akkoma.dev/stable/administration/backup/#restoremove&quot;&gt;Akkoma database restore/move&lt;/a&gt;
docs and ended up using psql, again under the postgres user, to run
&lt;code&gt;CREATE USER akkoma WITH ENCRYPTED PASSWORD &apos;&amp;lt;database-password&amp;gt;&apos;;&lt;/code&gt; and
&lt;code&gt;CREATE DATABASE akkoma OWNER akkoma;&lt;/code&gt;. &lt;i&gt;(well, i replaced akkoma with pleroma and later used alter queries to change them, but that&apos;s because my database is old)&lt;/i&gt;
&lt;/p&gt;
&lt;p&gt;
After that was ready, I used my firewall of choice (ufw) to
allow the servers to talk using their private IPs &lt;i&gt;(yay same datacenter)&lt;/i&gt;. After that was done, I ran
this command &lt;code&gt;pg_dump -U akkoma -C akkoma | ssh dynamo &quot;sudo psql -U akkoma -d akkoma&quot;&lt;/code&gt;
and waited.
&lt;i&gt;dynamo&lt;/i&gt; being the host of the new postgres server and owner of a spot in my .ssh/config.
&lt;/p&gt;
&lt;p&gt;
A Note:&lt;br/&gt;
you can directly do &lt;code&gt;pg_dump ... | psql ...&lt;/code&gt; but the Postgres upgrade
docs say you need to use the new psql version to upgrade, and the old server was missing that
binary. Instead of seeing if psql 13 would work or if I could get psql 15 working there, I
pipped it over ssh.
&lt;/p&gt;
&lt;p&gt;
It completed quicker than I thought, the command only took 21 minutes!, and all seemed well.
&lt;/p&gt;
&lt;h3 id=&quot;all-was-not-well&quot;&gt;All Was Not Well&lt;/h3&gt;
&lt;p&gt;
First, to prevent Akkoma from receiving activites that may
be lost if I have to revert, I disallowed everything on 80/443
except to my own IP so I could see if the web interface was working.
Yeah my website&apos;d be down for a bit but it was whatever. &lt;i&gt;(i think i could&apos;ve
	edited the nginx config to the same effect, but this was easier)&lt;/i&gt;
&lt;/p&gt;
&lt;p&gt;
I edited my &lt;code&gt;/etc/pleroma/config.exs&lt;/code&gt; to point
to the new postgres server and started Akkoma, but new-Postgres didn&apos;t
see a connection? Oh, I edited the wrong config and it was still
connecting to the local Postgres.
&lt;/p&gt;
&lt;p&gt;
I deleted &lt;code&gt;/etc/pleroma&lt;/code&gt;, so I&apos;d stop getting confused by
it, and edited the &lt;i&gt;correct&lt;/i&gt; file: &lt;code&gt;/opt/pleroma/config/prod.secret.exs&lt;/code&gt;
&lt;i&gt;(this is because I&apos;m a From Source install)&lt;/i&gt;.
&lt;/p&gt;
&lt;p&gt;
Aaaand it didn&apos;t work. Turns out it was trying to connect to it&apos;s own private IP
because copy-paste can be hard sometimes. Glad I stopped old-Postgres.
&lt;/p&gt;
&lt;p&gt;
Fixing that, I finally saw connections on the other machine. New problem: Akkoma
timesout the query after 15000ms (15 seconds) because it was taking too long. what?
and nothing is loading? ahhh.
&lt;/p&gt;
&lt;p&gt;
per the Akkoma docs from earlier, I ran some commands to try and cleanup
the database. I&apos;m a
From Source install, so I can &lt;code&gt;mix pleroma.database vacuum analyze&lt;/code&gt;
which did &lt;i&gt;not help&lt;/i&gt; so I tried it again with &lt;code&gt;full&lt;/code&gt; instead
of &lt;code&gt;analyze&lt;/code&gt;. This also did not help.
&lt;/p&gt;
&lt;p&gt;
I think what I was looking for was Akkoma to throw a fit as evidence that
something weird happened during the transfer, but nothing went wrong.
&lt;/p&gt;
&lt;p&gt;
So I was out of ideas. I am a Postgres novice and I&apos;m out of luck. What
does someone like me do when out of luck? Past the error into Google of course!
Maybe I should&apos;ve done that from the start, right, but I don&apos;t get
many results for Akkoma or Pleroma normally.
&lt;/p&gt;
&lt;p&gt;
So to google I went! And pasted &lt;q&gt;timed out because it queued and checked out the connection for longer than 15000ms&lt;/q&gt;
&lt;/p&gt;
&lt;p&gt;
and then I read
&lt;a href=&quot;https://elixirforum.com/t/timed-out-because-it-queued-and-checked-out-the-connection-for-longer-than-15000ms/34793/4&quot;&gt;a comment from al2o3cr&lt;/a&gt; that said:
&lt;/p&gt;
&lt;blockquote&gt;
	&lt;p&gt;Usually that&apos;s an indication of database issues, from missing indexes to queries that need optimization.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;
&quot;Missing indexes&quot; there caught my eye. It made a lot of sense to me. It&apos;s
taking so long because it&apos;s either digging through the 2.5 million activities
in the database, or it&apos;s trying to reindex the thing &lt;i&gt;(both?)&lt;/i&gt;. A quick
google later and I ran &lt;code&gt;REINDEX akkoma;&lt;/code&gt; from psql which literally
fixed all of my problems.
&lt;/p&gt;
&lt;p&gt;
That&apos;s it! take care and don&apos;t forget to reindex after your migration.
&lt;/p&gt;
		</content>
	</entry>
	
</feed>