[{"content":"Hey there, welcome to my blog! You\u0026rsquo;re in for a whirlwind of my adventures and insights. I hope you find it both useful and entertaining.\n","date":"2 June 2026","externalUrl":null,"permalink":"/","section":"","summary":"","title":"","type":"page"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/tags/aws/","section":"Tags","summary":"","title":"Aws","type":"tags"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/tags/cloud/","section":"Tags","summary":"","title":"Cloud","type":"tags"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/tags/python/","section":"Tags","summary":"","title":"Python","type":"tags"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/tags/react/","section":"Tags","summary":"","title":"React","type":"tags"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/tags/self-hosting/","section":"Tags","summary":"","title":"Self-Hosting","type":"tags"},{"content":" Syncify 2 # Many moons have passed since I last touched Syncify. Finally, with a little more time on my hands (this blog post was due in January, oops) and some help from a couple of friends, we\u0026rsquo;ve got a new version out!\nSyncify will now sync your liked songs in the background for you! 🎉🎉🎉\nThe tech stack has also changed to Python (FastAPI) + React + AWS. Quite the difference to the old stack which was Golang + HTMX.\nPython was my friends\u0026rsquo; preferred language and was also helpful as a low-stress way to learn more about Python as a backend language, using Syncify as a testbed for all the things I did at DreamDesk AI.\nNew UI # With the introduction of background jobs, we no longer need to force the user to sit on the page as the backend runs your job in one request. Instead, the frontend now just allows you to queue a sync job and then shows you the progress. It also now lists some of your past syncs.\nThe old version was cleaner for sure, but it was really minimal. You didn\u0026rsquo;t even get a progress indicator. Even this version is quite rough around the edges, I probably need to remove the sync request ids and figure out a better layout for the info and buttons we\u0026rsquo;ve just lazily thrown at the top right.\nIn terms of API access, we\u0026rsquo;re now using TanStack Query. If you\u0026rsquo;ve not used it, you\u0026rsquo;re not doing frontend right. It\u0026rsquo;s one of the best abstractions I\u0026rsquo;ve ever seen. It essentially worries about a load of state management around a fetch query that you\u0026rsquo;d otherwise have to duplicate for every endpoint you have. Need to show something specific if the query errored out? {query.error \u0026amp;\u0026amp; \u0026lt;div\u0026gt;Something went wrong\u0026lt;/div\u0026gt;}. Need the query to refetch automatically? { ..., refetchInterval: 5000, }. That\u0026rsquo;s just scratching the surface too. It\u0026rsquo;s lovely.\nAll that said, there\u0026rsquo;s one thing I don\u0026rsquo;t like about the way we currently show progress updates (this isn\u0026rsquo;t a fault of TanStack Query just to be clear). When showing progress updates, we\u0026rsquo;re using that re-fetch interval to poll the backend for the current status of the sync request. This isn\u0026rsquo;t particularly efficient, and I\u0026rsquo;d much rather have the backend push updates to the frontend using something like Server Sent Events (SSE). This would add complexity, and we wouldn\u0026rsquo;t be able to use TanStack Query (boo), but ultimately the repeated re-fetch isn\u0026rsquo;t particularly scalable.\nRight now, we refetch every 3 seconds if the initial jobs request contains any incomplete jobs. An SSE implementation instead, would open an SSE request to the backend which would subscribe to updates to a job\u0026rsquo;s progress. This would allow us to update the frontend in real-time as the job progresses without the need for repeated polling. There are still a few challenges with this, though, as we\u0026rsquo;ll discuss soon.\nJob Queues \u0026amp; Idempotency # With the new background syncing system came some interesting problems to solve, the biggest of which is scheduling the jobs. There were many ways I could have gone about implementing this, my first thought went to something like Celery, but I really didn\u0026rsquo;t want to add another dependency. For the first iteration, I decided on creating a Postgres table that could act as a queue.\nThe table ended up looking like this:\ncreate table sync_requests ( id serial primary key, user_id varchar not null references users on delete cascade, song_count integer not null, progress double precision not null, created timestamp with time zone default now() not null, completed timestamp with time zone ); create index ix_sync_requests_user_id on sync_requests (user_id); create index ix_sync_requests_completed on sync_requests (completed); create index ix_sync_requests_created on sync_requests (created); When a user requested a sync, or the scheduler ran and created a sync job on behalf of a user, it got placed in this table.\nA worker then pulled off the queue using the following query:\nselect * from sync_requests where completed is null order by id asc limit 1; The completed column acted as the idempotency key, so that we weren\u0026rsquo;t running the same job twice. The one thing this system didn\u0026rsquo;t handle was the worker crashing mid-job. I\u0026rsquo;d accepted that tradeoff initially since there\u0026rsquo;s always another job for each user scheduled soon after, but it was a known rough edge.\nAWS # Recently I decided I wanted to move the service off of my home server and onto AWS. The poor little NUC under my stairs isn\u0026rsquo;t particularly brilliant for user-facing applications in terms of uptime and reliability. The move meant I could set up some Infrastructure as Code (IaC) and repeatably deploy Syncify into AWS and utilise more cloud native serverless services like Simple Queue Service (SQS) and DynamoDB and still pay next to zero.\nSo SQS and DynamoDB replaced the postgres instance we just discussed. We got better idempotency and error handling with dead letter queues with super simple alerts using CloudWatch and SNS. SQS now handles the job queue, and DynamoDB handles the job status, which is still polled from the frontend via the API.\nSpeaking of the API, it\u0026rsquo;s now one Lambda function behind a basic API Gateway. Super simple and it works really well. The worker for the background syncing is also a Lambda too, it pulls straight off the SQS queue and updates job statuses in DynamoDB.\nSync Job Progress Updates # As we mentioned above, the frontend polls for job progress every few seconds, which burns a lot of Lambda requests and isn\u0026rsquo;t scalable. (AWS would happily do it with its infinite scale serverless infrastructure\u0026hellip; might just cost a small fortune.) It looks like this right now:\nsequenceDiagram participant FE as Frontend participant GW as API Gateway participant API as API Lambda participant DDB as DynamoDB participant SQS as SQS Queue participant W as Worker Lambda FE-\u003e\u003eGW: POST /sync (queue job) GW-\u003e\u003eAPI: invoke API-\u003e\u003eDDB: put job (status=pending) API-\u003e\u003eSQS: send message API--\u003e\u003eFE: 202 Accepted SQS-\u003e\u003eW: deliver message activate W W-\u003e\u003eDDB: update progress W-\u003e\u003eDDB: update progress W-\u003e\u003eDDB: status=complete deactivate W loop every 3s while job incomplete FE-\u003e\u003eGW: GET /sync/{id} GW-\u003e\u003eAPI: invoke API-\u003e\u003eDDB: read job API--\u003e\u003eFE: job status + progress end The SSE alternative I\u0026rsquo;m thinking of is to have the worker push events onto an EventBridge bus and then have a streaming Lambda subscribe to the bus. It might look something like this:\nsequenceDiagram participant FE as Frontend participant URL as Function URL participant SSE as SSE Lambda participant EB as EventBridge participant W as Worker Lambda participant DDB as DynamoDB FE-\u003e\u003eURL: GET /sync/{id}/stream URL-\u003e\u003eSSE: invoke (response streaming) activate SSE SSE-\u003e\u003eEB: subscribe to job {id} W-\u003e\u003eEB: put event (progress 25%) EB--\u003e\u003eSSE: deliver event SSE--\u003e\u003eFE: data: progress 25% W-\u003e\u003eEB: put event (progress 75%) EB--\u003e\u003eSSE: deliver event SSE--\u003e\u003eFE: data: progress 75% W-\u003e\u003eDDB: status=complete W-\u003e\u003eEB: put event (complete) EB--\u003e\u003eSSE: deliver event SSE--\u003e\u003eFE: data: complete deactivate SSE This would significantly reduce the number of Lambda invocations. I\u0026rsquo;ve also removed API gateway since it has some limitations around timeouts. That said, I\u0026rsquo;m not entirely sure if it\u0026rsquo;s worth it because those streaming lambdas will run for a lot longer than each Lambda invocation. It\u0026rsquo;s a trade-off between the number of invocations and the duration of each invocation. Until it\u0026rsquo;s actually a problem we can prove with data, I probably won\u0026rsquo;t bother with the extra complexity.\nScheduling # For each user that creates an account, I am creating an EventBridge Schedule that runs every 24 hours that adds a sync job to the SQS on their behalf. I had to be a little careful about leaving orphaned schedules behind if a user deletes their account or their refresh token gets revoked. This is far less complexity than the separate python job I had running previously. It really didn\u0026rsquo;t work very well\u0026hellip; glad to be rid of it.\nMigration # Each of the users that had an account on the NUC was gathered out of a pg_dump using an AI-generated script (god, I love AI) and then both inserted into the database and had their EventBridge Schedule created. Hopefully, all your syncs are working :)\nFuture Features # I\u0026rsquo;d love to add more features to Syncify. Let me know if you like any of these ideas or have any of your own! Search for an issue on the GitHub repository and give it a thumbs up or suggest a new feature.\nAI playlist generation (e.g. \u0026ldquo;make me a playlist of songs that I haven\u0026rsquo;t listened to in a while\u0026rdquo; or \u0026ldquo;move jazz songs into their own playlist\u0026rdquo;) Cool analytics like how many songs you have over time Maybe song deduplication (e.g. explicit/non-explicit versions of the same song) Perhaps a way to sync your liked songs across platforms (YouTube, Apple Music, etc.), even just a csv export Bidirectional sync (from your Syncify playlist(s) to your Liked Songs playlist) Friend graphs, common artists/songs\u0026hellip; probably a stretch My Friends Helped # Thanks to Paul and Yu who chipped in to update the tool to handle background syncing and better playlist management :)\nI hope you enjoy Syncify!\n","date":"2 June 2026","externalUrl":null,"permalink":"/posts/syncify2/","section":"Posts","summary":"New features, new tech stack.","title":"Syncify 2","type":"posts"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"30 March 2025","externalUrl":null,"permalink":"/tags/ai/","section":"Tags","summary":"","title":"Ai","type":"tags"},{"content":" Context # I\u0026rsquo;ve recently started working for a new company as a DevOps Engineer 🥳. We use Python exclusively for our projects here; Picking it up again has been an\u0026hellip; interesting experience. I am reminded of how little fun I have when getting Python to work for anything beyond scripting. That\u0026rsquo;s a rant for another time, though.\nSoon after I joined the company, I was tasked with introducing separate deployment environments; production, staging, etc. At the time, they had one database instance handling the live service and their development work simultaneously. Not the best idea to let your engineers change the schema of the live app.\nAs I was working on creating separate databases for each environment, I remembered that changes made to a schema in a development environment would not be reflected in a production environment without some manual work, which is always prone to error. In order to avoid this, I went on the hunt for a migration management tool.\nAlembic # I came across a tool called Alembic. It appears to be the standard choice for handling database migrations in Python, like the .Net Entity Framework might be for C#. Unfortunately, the app was using raw SQL with pymysql, the low-level database driver for MySQL databases. We\u0026rsquo;d be missing out on half of Alembic\u0026rsquo;s functionality if we stuck with what we had. Thankfully, we had a single file with all the database queries written in raw SQL and there wasn\u0026rsquo;t that many. So we decided to tweak our approach to database logic early.\nSQLAlchemy # Alembic is designed to use SQLAlchemy under the hood to achieve things like automatic migration generation. Based on minimal internet browsing, SQLAlchemy appears to be the de facto option for Python database work, which makes sense given the Alembic integration. As such I had us adopt and migrate our codebase to it.\nSQLAlchemy has 2 usage patterns:\nCore # The first component, \u0026ldquo;Core\u0026rdquo;, is a SQL wrapper of sorts where queries look like:\nwith engine.connect() as conn: result = conn.execute(text(\u0026#34;SELECT x, y FROM some_table WHERE y \u0026gt; :y\u0026#34;), {\u0026#34;y\u0026#34;: 2}) for row in result: print(f\u0026#34;x: {row.x} y: {row.y}\u0026#34;) Very simple, we just execute raw SQL with some fancy parameter insertion done for us.\nORM # The ORM portion (what we\u0026rsquo;re using at my new company) allows us to specify almost everything in a Pythonic manner such as our schema:\n# https://docs.sqlalchemy.org/en/20/tutorial/index.html from typing import Optional from sqlalchemy.orm import Mapped from sqlalchemy.orm import mapped_column class User(Base): __tablename__ = \u0026#34;user_account\u0026#34; id: Mapped[int] = mapped_column(primary_key=True) name: Mapped[str] = mapped_column(String(30)) fullname: Mapped[Optional[str]] def __repr__(self) -\u0026gt; str: return f\u0026#34;User(id={self.id!r}, name={self.name!r}, fullname={self.fullname!r})\u0026#34; and our queries:\n# https://docs.sqlalchemy.org/en/20/tutorial/index.html from sqlalchemy import select stmt = select(User).where(User.name == \u0026#34;spongebob\u0026#34;) with Session(engine) as session: for row in session.execute(stmt): print(row) # (User(id=1, name=\u0026#39;spongebob\u0026#39;, fullname=\u0026#39;Spongebob Squarepants\u0026#39;),) While I enjoy raw SQL, this seamless, Pythonic way of interacting with the database is a joy to work with.\nWe opted for the ORM option since it allowed us to declaratively write our database schema and have Alembic automatically generate migrations when we changed it.\nLLMs Make Us FAST # As with almost all software engineers nowadays, rewriting our queries was a breeze. Simply do a little prompt tweaking with a sample database query:\nHey ChatGPT! Could you pretty please translate this SQL query into SQLAlchemy ORM language for me. Please and thank you. You\u0026rsquo;re the best btw ;)\nThen feed it the rest of the file and copy pasta into your code\u0026hellip;\nLastly, test it? Then again, LLMs are always right so maybe there\u0026rsquo;s no need to waste more time 😂💀.\nTo be fair it worked quite well. Only some minor tweaks required to tidy up after ChatGPT.\nWhere\u0026rsquo;s the \u0026ldquo;But\u0026rdquo;? # Here\u0026rsquo;s the thing, LLMs are great for things that don\u0026rsquo;t change.\nHey ChatGPT, old chum! Could you please explain to me why the sky is blue?\nThis is a great prompt. Unless physics is upturned tomorrow, the LLM will be correct in its answer.\nCode on the other hand, boy does that change FAST.\nAI is Only as Good as its Dataset # As it turns out, SQLAlchemy had a major release sometime between when ChatGPT 3.5 was trained and when I asked it to translate my SQL queries.\nEverything it wrote was out of date by a mile. The only reason it worked is because the SQLAlchemy team managed to maintain near perfect backwards compatibility. Unfortunately \u0026ldquo;near\u0026rdquo; is not enough.\nRecently, I went to the up-to-date docs to write a specific type of query (I needed some weird join syntax). Lo and behold, I found my copy pasta\u0026rsquo;d example spitting errors back at me.\nI trawled through stack traces for hours trying to find where I\u0026rsquo;d screwed up.\nI also searched the docs but half the time, Google brings up the outdated docs too! I guess it\u0026rsquo;s an age old problem 🤷‍♂️.\nFinally I found it, the docs for creating a schema. Was it completely different to the old way? No. But was it using a completely different function to create relationships? Yes, yes it was.\nConclusion # In the end, I don\u0026rsquo;t know how much time the LLM saved me. It did speed up the initial implementation by a huge factor, I have no doubt about that. However, the time I had to spend debugging and then manually updating the schema code? Likely about the same as if I had just checked the docs in the first place.\nThat said, perhaps if I had more closely validated what the LLM was spitting out\u0026hellip; Perhaps cross referencing the docs and actually understanding the code that I was writing\u0026hellip;\nNah, AI go brrrr.\n","date":"30 March 2025","externalUrl":null,"permalink":"/posts/outdated-ai/","section":"Posts","summary":"Using an LLM as a software engineer just means you’re writing outdated code.","title":"AI is Outdated","type":"posts"},{"content":"","date":"30 March 2025","externalUrl":null,"permalink":"/tags/database/","section":"Tags","summary":"","title":"Database","type":"tags"},{"content":"","date":"30 March 2025","externalUrl":null,"permalink":"/tags/llm/","section":"Tags","summary":"","title":"Llm","type":"tags"},{"content":"","date":"21 March 2025","externalUrl":null,"permalink":"/tags/oauth/","section":"Tags","summary":"","title":"Oauth","type":"tags"},{"content":" Preface # I have been writing this since before my Kitchen Debt post. It has taken far too long, so please, enjoy to the fullest extent possible 🥲.\nA Short (I Lie) History Lesson # For longer than I\u0026rsquo;ve been alive, programmers have been implementing password authentication into their services and applications. Unfortunately, over the years passwords have to proven to be rather insecure and inconvenient;\nThe title of the most notorious password dump in history likely goes to the RockYou dump of 2009. It contains 14 341 564 unique passwords spread across 32 603 388 accounts, all of which were stored in plain text. That\u0026rsquo;s a staggering 139.92 MB of just passwords.\nThe RockYou dump provided researchers and hackers alike a window into human habits and traits when it comes to crafting a password. The dump is so useful to this day, that Kali Linux (the Linux distribution designed for hacking), ships the dump pre-packaged in the OS.\nHumans # Let\u0026rsquo;s talk about those human habits; One of the most common things we do with passwords is reuse them on different websites (yes it still counts as re-use if you change the number on the end). Jack Rhysider published an episode of the Darknet Diaries podcast (highly recommend), in which he talks to someone who was able to steal unreleased music from artists partially because they re-used their passwords on different websites. One company gets hacked, and suddenly the attackers have access to all your accounts, everywhere.\nAnother poor security tactic of humans is creating passwords that relate to things they\u0026rsquo;re unlikely to forget such as a pet\u0026rsquo;s name or a significant date. I\u0026rsquo;m looking at you, Mr fluffy1995. All it takes is for someone to look at your Instagram (where you undoubtedly worship your pet and have your age in your bio) and then have a few educated guesses at your password.\nPassword Storage # Thankfully nowadays, (most) companies aren\u0026rsquo;t storing your passwords in plaintext. The new standards involve salting and hashing your password. What does that look like? Well let\u0026rsquo;s start with a password like D0YouH4veLigma?:\nThe first step is to salt the password, ideally pseudo-randomly. This might end up looking like 8D0YhouH54ve5Lig!ma?v, here we\u0026rsquo;ve interspersed some pseudo-random characters (generated with a seed like the user\u0026rsquo;s ID, basically something that we can recreate at every login). This makes sure the password isn\u0026rsquo;t immediately guessable.\nThe next step is to hash the password, an (ideally) one way transformation to a standard length output. One of the earlier widely used hashing algorithms was MD5, unfortunately it was and is quite easy to break; The likes of SHA-256 have now superseded it. Hashing our salted password with SHA-256 gives us 03e38186e2037be20bffb263869d6332b3aa42b2349d09aff1a5c7ad5af7a3da.\nNow even if an attacker gets the database, they could likely never figure out what your original password was.\nWhile I\u0026rsquo;ve just explained how this works in principle, the method above still isn\u0026rsquo;t secure. Don\u0026rsquo;t roll this stuff yourself; just use something like bcrypt or argon2id.\nBrute Force # Rather than getting at a leaked database of passwords, attackers can always still try the front door by guessing your password. This can be done with tools like Hydra. This is where password complexity rules come in. The less complex a password is, the faster someone can iterate through all the possible password combinations and happen across it.\nMany companies implement password rules for this reason. The most effective one against brute force attacks is sheer length of password; Something above 12 characters would simply take too long to brute force. One of the least effective rules is forcing users to change their password every X months. I guarantee you 99% of employees are just changing one number in their password every time. This means that if any of those passwords leak, it takes the attacker less than a second to find your current password. Plus, it\u0026rsquo;s generally really annoying. So if you\u0026rsquo;re a system administrator:\nOpen Authorization (OAuth) # Now that I\u0026rsquo;ve called out passwords for being rather terrible, let\u0026rsquo;s look at the main topic of conversation.\nI look at OAuth as a method of Single Sign On (SSO). As the name implies, you use a Single method of authentication to access all your services from any of your devices. Fewer passwords, yay!\nA great example of OAuth and SSO is Google. Ever noticed how almost every website and app allows you to login using your Google account? That\u0026rsquo;s SSO. Each app accepts your Google account as an account on its system which means you only need to remember the credentials for your Google Account. One password for Google, and that\u0026rsquo;s it.\nIn a nutshell, OAuth allows computer systems to authorize each other instead of relying on a human entering human passwords to authenticate themselves.\nDefinitions # There\u0026rsquo;s a few definitions that will help us understand the rest of this post:\nClient - The application that is attempting to get access to the user\u0026rsquo;s account. E.g. YouTube/Spotify/LinkedIn Resource Server - The API/Server that serves access to a user\u0026rsquo;s information. E.g. Google User Info Endpoint Authorization Server - This is the IDP that provides the user with an authorization interface. E.g. Google Auth Resource Owner - The user. E.g. You 😉 Access Tokens # Access tokens are the bread and butter of OAuth. They are what get given to the client to prove that it is authorized to access something on a resource server.\nAccording to the OAuth spec, access tokens:\nare generated and managed by the authorization server must be kept confidential and secure (only seen by the authorization server and the resource server) can be any format (as long as they can be stringified) Grant Types # Grant Types (also referred to as Flows) are methods by which a client can acquire an access token.\nAuthorization Code + PKCE # The authorization code flow involves a temporary credential (code 🤯) issued by the authorization server to a client application after a user successfully authenticates and grants consent.\nIt serves as an intermediate step before obtaining an access token.\nsequenceDiagram participant User participant Client participant Authorization Server participant Resource Server User-\u003e\u003eClient: Initiates Authorization Request Note over Client: Generate Code Verifier, Code Challenge \u0026 State Client-\u003e\u003eAuthorization Server: Authorization Request (with Code Challenge \u0026 State) Authorization Server-\u003e\u003eUser: Request User Approval User-\u003e\u003eAuthorization Server: Approves Request Authorization Server-\u003e\u003eClient: Authorization Code (includes State) Note over Client: Verify State, Exchange Code for Token (with Code Verifier) Client-\u003e\u003eAuthorization Server: Token Request (with Code Verifier) Authorization Server-\u003e\u003eClient: Access Token Note over Client: Access Protected Resource Client-\u003e\u003eResource Server: Request Resource (with Access Token) Resource Server-\u003e\u003eClient: Protected Resource As of OAuth 2.1, PKCE is a required component of an Authorization Code flow, hence we\u0026rsquo;re not going to talk about them separately. Just know that you will probably find auth code flows without code challenges/verifiers in the wild. Without PKCE, a client secret is always required.\nClient Credentials # This flow is more for machine-to-machine communication; often replacing something simple like a periodic data fetch from an API.\nsequenceDiagram participant Client as Client participant AuthServer as Authorization Server participant ResourceServer as Resource Server Client-\u003e\u003eAuthServer: Request access token (with client id and secret) Note over AuthServer: Validate client credentials AuthServer-\u003e\u003eClient: Return access token Client-\u003e\u003eResourceServer: API Request with Access Token Note over ResourceServer: Validate access token ResourceServer-\u003e\u003eClient: Return resource data This might look like basic auth with extra steps, but it does actually provide some benefits if your use-case warrants the complexity overhead:\nGranular access control via scopes Short-lived credentials with (almost) built-in rotation Better integration with things JWTs for stateless auth Device Code # This flow is used by devices or applications with limited input capabilities like IoT devices. It allows users to authenticate and authorize the application on a secondary device, typically a mobile phone or desktop computer, by using a unique code displayed on the device.\nsequenceDiagram participant Client as Device participant User participant AuthorizationServer participant ResourceServer Client-\u003e\u003eAuthorizationServer: Request device code and user code AuthorizationServer-\u003e\u003eClient: Return device code and user code Client-\u003e\u003eUser: Display user code and URL User-\u003e\u003eUserDevice: Open URL and enter user code UserDevice-\u003e\u003eAuthorizationServer: User enters credentials and authorizes AuthorizationServer-\u003e\u003eUserDevice: User authenticated and authorized Client-\u003e\u003eAuthorizationServer: Polling for access token AuthorizationServer-\u003e\u003eClient: Access token granted Client-\u003e\u003eResourceServer: Access resources with token ResourceServer-\u003e\u003eClient: Return requested resources I actually think this might be a good way to handle mobile app authentication instead of using weird webview integrations but I have yet to prove out my theory. Reach out if you know how it should work.\nRefresh Token # Remember how we\u0026rsquo;ve said so far that access tokens are short lived? Well I think we can both agree that needing to log into your authentication provider every 5 minutes would be a terrible user experience.\nThankfully this is solved using refresh tokens. When performing an auth code flow, the authorization server can optionally also return a refresh token.\nUsing the refresh token, you can (only) perform a refresh token flow to get a new access token to replace an expired one; all without any user interaction.\nsequenceDiagram participant U as User participant C as Client participant AS as Authorization Server participant RS as Resource Server Note over C: Access token expires U-\u003e\u003eC: Initiate request for data C-\u003e\u003eAS: Send refresh token Note over AS: Validate refresh token AS-\u003e\u003eC: Issue new access token (and potentially new refresh token) C-\u003e\u003eRS: API request with new access token Note over RS: Validate access token RS-\u003e\u003eC: API response C-\u003e\u003eU: Display response for user Notice how the user doesn\u0026rsquo;t have to interact with the Authorization server this time! Very slick 😉.\nPassword Grant (Deprecated) # This deprecated flow essentially just gets the client to pass the user\u0026rsquo;s username and password to the authorization server to get an access token.\nThis was deprecated because, as you can imagine, it\u0026rsquo;s not the best idea to let the client handle the users actual credentials in case it is malicious.\nDo not use this flow if you can help it. I\u0026rsquo;ve really wanted to in the past for things like my work on implementing OAuth for Docker Mailserver but it\u0026rsquo;s just not the way.\nOpenID Connect (OIDC) # OIDC is an extension of OAuth 2.0 that adds authentication capabilities to the protocol (where OAuth 2.0 focuses on authorization).\nIdentity is a core component of authentication (proving the user is who they say they are), and is introduced by the concept of an ID Token, which is a JSON Web Token (JWT) that contains information about the authenticated user. This token is issued alongside the access token in the OAuth flows shown earlier.\nAlongside the ID Token, OIDC adds a UserInfo Endpoint that provides a client with similar data to what\u0026rsquo;s in the ID Token.\nOIDC also introduces a quality of life discovery mechanism. This allows clients to find all the endpoints and features provided by an auth provider using just one standardized URL (example.com/.well-known/openid-configuration).\nJavascript Web Tokens (JWTs) # I should probably explain what JWTs are since I mentioned them in the previous section.\nA JWT is a base64-encoded JSON object, which contains a header and a payload and optionally a signature. The header contains information required to decode the payload and verify the signature. The payload contains the\u0026hellip; data we actually care about.\nLet\u0026rsquo;s look an example one:\neyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvZU1hbWEiLCJpYXQiOjE1MTYyMzkwMjJ9.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c So as you can see, all the information you could possibly need in an ID token is contained within the string above. I\u0026rsquo;m joking don\u0026rsquo;t worry 😂.\nWe can actually glean a little information from the base64 encoded string above though. We can see the three distinct sections, separated by periods.\nHeader # { \u0026#34;alg\u0026#34;: \u0026#34;HS256\u0026#34;, \u0026#34;typ\u0026#34;: \u0026#34;JWT\u0026#34; } The header is a JSON object containing a signing algorithm (alg) and a type (typ) which in our case is always going to be \u0026quot;JWT\u0026quot;.\nPayload # { \u0026#34;sub\u0026#34;: \u0026#34;1234567890\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;Joe Mama\u0026#34;, \u0026#34;iat\u0026#34;: 1516239022 } The payload contains the \u0026ldquo;claims\u0026rdquo; that we ask for when we send our list of scopes to the authorization server.\nYou can find a list of standard claims on the openid website.\nSignature # SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c This is the only part that\u0026rsquo;s not JSON, it\u0026rsquo;s generated using the algorithm from the header, and it should be verifiable by the client.\nIn this example, the token is using HS256 a symmetric signing algorithm.\nThis means the server and the client share a key that they pass to the algorithm along with the payload contents.\nPerforming the signature generation will spit out the contents of the signature. Unless there\u0026rsquo;s some naughty stuff going on in the middle, both sides will always generate the same signature.\nAnother option for the signature is RS256 which is asymmetric; The server will first use a private key to generate the hash of the payload. Then the client grabs the server\u0026rsquo;s public key and verifies the signature. The algorithm underneath RS256 is RSA. Let\u0026rsquo;s get into how it works.\nRSA # Over the years, we\u0026rsquo;ve been blessed with algorithms that can provide near-perfect security. The example most people will know is RSA which has been widely used for decades at this point and (at a sufficient key length) is considered perfectly secure today.\nRSA is a \u0026ldquo;public-key cryptosystem\u0026rdquo;\u0026hellip; yeah I don\u0026rsquo;t like Wikipedia drool either. In human terms; RSA utilizes 2 keys, a public and private key, to encrypt and decrypt data using some clever one-way mathematics. I say one-way because trying to break the encryption without the other key is effectively impossible, especially at large key lengths.\nFor example, here\u0026rsquo;s a private key:\n-----BEGIN RSA PRIVATE KEY----- MIIBOgIBAAJBAKj34GkxFhD90vcNLYLInFEX6Ppy1tPf9Cnzj4p4WGeKLs1Pt8Qu KUpRKfFLfRYC9AIKjbJTWit+CqvjWYzvQwECAwEAAQJAIJLixBy2qpFoS4DSmoEm o3qGy0t6z09AIJtH+5OeRV1be+N4cDYJKffGzDa88vQENZiRm0GRq6a+HPGQMd2k TQIhAKMSvzIBnni7ot/OSie2TmJLY4SwTQAevXysE2RbFDYdAiEBCUEaRQnMnbp7 9mxDXDf6AU0cN/RPBjb9qSHDcWZHGzUCIG2Es59z8ugGrDY+pxLQnwfotadxd+Uy v/Ow5T0q5gIJAiEAyS4RaI9YG8EWx/2w0T67ZUVAw8eOMB6BIUg0Xcu+3okCIBOs /5OiPgoTdSy7bcF9IGpSE8ZgGKzgYQVZeN97YE00 -----END RSA PRIVATE KEY----- and a public key:\n-----BEGIN RSA PUBLIC KEY----- MEgCQQCo9+BpMRYQ/dL3DS2CyJxRF+j6ctbT3/Qp84+KeFhnii7NT7fELilKUSnx S30WAvQCCo2yU1orfgqr41mM70MBAgMBAAE= -----END RSA PUBLIC KEY----- As the names suggest, you are supposed to keep your private key to yourself, but you would usually share your public key. You can use either key to encrypt data, but they are used for different things.\nLet\u0026rsquo;s say I have my private key and I hand my public key to my friend Donald (Duck).\nThe typical use-case for RSA is for Donald to send me a secure message. To do this, he would write his message and encrypt it with my public key. He would then send this encrypted message to me. Anyone who tries to read it along the way cannot because it is encrypted. I would then use my private key (that I\u0026rsquo;ve hopefully not leaked) to decrypt and read Donald\u0026rsquo;s message.\nThe other use-case would be if I\u0026rsquo;ve sent a message to Donald, and he wants to be sure it came from me. In this scenario, I would encrypt my message with my private key before sending it to him. Donald would be able to decrypt it with my public key only if the message was encrypted with my private key. This implies that the message did indeed, come from me. (Assuming again, that I haven\u0026rsquo;t accidentally leaked my private key)\nSo what use-case is suited to RS256?\nThat\u0026rsquo;s right, the second one! It allows us to verify the signature came from who we expect.\nThat\u0026rsquo;s All Folks # Congratulations or I\u0026rsquo;m sorry, somehow you\u0026rsquo;ve made it this far\u0026hellip; you either love security or need to touch grass.\nEither way, next time you click ‘Sign in with Google,’ you’ll know exactly what’s happening under the hood.\nAnd remember, passwords bad, use a password manager (randomly generated per website) if you have to use them, otherwise I hear passkeys are in fashion at the moment.\n","date":"21 March 2025","externalUrl":null,"permalink":"/posts/oauth/","section":"Posts","summary":"The inner workings of OAuth and OIDC.","title":"OAuth \u0026 Modern Authentication","type":"posts"},{"content":"","date":"21 March 2025","externalUrl":null,"permalink":"/tags/oidc/","section":"Tags","summary":"","title":"Oidc","type":"tags"},{"content":"","date":"5 October 2024","externalUrl":null,"permalink":"/tags/human-interaction/","section":"Tags","summary":"","title":"Human Interaction","type":"tags"},{"content":" The Predicament # Recall a time when your manager (non-tech stakeholder) wants a feature done muy rapido (very fast), and you\u0026rsquo;ve got to mutilate your otherwise \u0026ldquo;perfect\u0026rdquo; codebase to do it. Never a fun time but nonetheless, we both know the possible solutions to this problem:\nSuccumb to the pressure and implement it as fast as possible, leaving a huge mess that will haunt the next poor sod who dares need to make a change in the future. Make your objections known, implement it quickly and vow to fix it afterwards™. You know full well that your manager will have another urgent feature ready to shove down your throat immediately after this one. Agree to implement it, but explicitly state that it will not be done in time. This little maneuver\u0026rsquo;s gonna cost you 2 years on your next promotion. Quit immediately, become a YouTuber, realise you can\u0026rsquo;t make videos, beg for your job back, implement the feature. None of these are ideal. These are the options because the root of the problem is your stakeholder\u0026rsquo;s lack of understanding of the situation. I\u0026rsquo;m reaching here, I know. But I came up with an analogy that I wanted to share.\nKitchen Debt # Usually I\u0026rsquo;m quite tidy in the kitchen. I have no cleaning up to do at the end because I clean as I go. It means I can eat, throw my bowl in the dishwasher, and not have someone yelling at me to clean up so that they can use the kitchen. It does take a little longer to work like this, but the food turns out better when I fuss over it and there\u0026rsquo;s no mess.\nI was rushing to make dinner yesterday (my famous ragù), because my friend called to go bouldering with an hour\u0026rsquo;s notice. Needless to say, my etiquette went straight out the window. I left onion peels on the counter, knives in precarious locations, and numerous unwashed pots/pans sprawled around the cooker. Having inhaled my food and put on my shoes to leave, my mother comes down. She takes one look at the kitchen and says \u0026ldquo;I\u0026rsquo;m about to make a cake, clean up your mess\u0026rdquo;. I love my mother dearly, and so I left for climbing. She was not happy.\nThis sounds a lot like tech debt to me. I needed to implement a feature (cook ragù) in a short period of time (bouldering in an hour), leaving technical debt (a mess). The next poor sod (my mum) needed to make a change (cook something else), and was haunted by it (had to clean up my mess first).\nTechnical Debt # Usually technical debt isn\u0026rsquo;t actually as clear-cut as I\u0026rsquo;ve made out. It\u0026rsquo;s often difficult to even realise that you\u0026rsquo;re creating it, especially the earlier you are in your career. Some might go to the extreme either way when dealing with technical debt:\nPerson 1 might decide the best course of action is to prevent any and all possible tech debt. This is the person who tries to account for every single possible use-case that could ever arise and could turn something as benign as a \u0026ldquo;Hello World\u0026rdquo; app into a nightmare that needs 3 on-call principal engineers at all times to maintain.\nPerson 2 implements things\u0026hellip; as fast as humanly possible. Ignore all best practices just get the thing working and ship it. They would tack a horse onto a horsefly if it made the horse\u0026hellip; fly. Unsurprisingly, this also ends up being a 3 on-call principal engineers nightmare.\nI\u0026rsquo;ve been on both sides of this curve. I\u0026rsquo;ve gone from being person 2 right at the beginning of my coding life, very quickly almost all the way to person 1 after I learned how annoying it was to change 10 instances of the same code that all do one thing slightly differently. Then transitioning back towards person 2 because implementing a generic-ridden, extensible, recursive iterator for 2 possible use-cases is a nightmare. Nowadays, I\u0026rsquo;m somewhere in the middle, and I\u0026rsquo;ve realised that I\u0026rsquo;ve started to abide by some rules that help me stay there.\nThe simpler something is, the easier it is to: debug, change, or rewrite. For you or someone else. Elegant is not simple. Implement it fast, copy it once, implement it \u0026ldquo;properly\u0026rdquo; (documented, easy to use, easy to understand, not repeated) the 3rd time and fix the first 2 while you\u0026rsquo;re there. Be conservative in your time estimates, you never know what kinds of problems will spring up. I go for 1.25x my first estimate. Conclusion # Code and associated technologies are a living thing, you can\u0026rsquo;t expect to get things right the first time or to never have to change anything. Hopefully this helps you deal with your managers and tech debt in general.\n","date":"5 October 2024","externalUrl":null,"permalink":"/posts/kitchen-debt/","section":"Posts","summary":"An analogy for tech debt for your non-tech peers.","title":"Kitchen Debt","type":"posts"},{"content":"","date":"20 July 2024","externalUrl":null,"permalink":"/tags/cgnat/","section":"Tags","summary":"","title":"Cgnat","type":"tags"},{"content":"","date":"20 July 2024","externalUrl":null,"permalink":"/tags/networking/","section":"Tags","summary":"","title":"Networking","type":"tags"},{"content":" What In The Seven Hells\u0026hellip;? # No, I haven\u0026rsquo;t finally lost my marbles.\n*shakes head vigorously*\nYep, still there.\nRemember how I said that you could probably link up a Cloud Gateway with your home router instead of straight to your server? Well, I finally racked up enough reasons to give the whole house a publicly routable IPv4 address instead of just my server.\nMy brother wanted to host a Minecraft server on his laptop 🙄. I need to re-setup a VPN to my internal network so my dad and I can reach our smart devices from the outside without exposing them publicly. This one pertains specifically to the title of this post 😉. It\u0026rsquo;s cooler to say \u0026ldquo;I gave my house another WAN address\u0026rdquo; than \u0026ldquo;I routed traffic to my server\u0026rdquo;. Why Didn\u0026rsquo;t You Just Use Tailscale? # I like Tailscale a lot. However, it limits the number of free users you can have. I.e. people with separate logins.\nEveryone that should have access to the VPN is already set up on an IDP and unless I pay the big bucks ($6/user/month at time of writing), I can\u0026rsquo;t connect them up. Plus it\u0026rsquo;s not self-hosted, which goes against the whole ethos with which I build all this cool stuff.\nI like Headscale even more, which is self-hosted, and would likely be a great option. One reason I didn\u0026rsquo;t opt for it is that hindsight is 20-20 😂. Plus I would still have wanted to host it locally which is basically the same as what I\u0026rsquo;ve ended up with anyway minus some bells and whistles. I might swap to Headscale in the future, we\u0026rsquo;ll see.\nWhat Did You Actually Build? # Firstly, I\u0026rsquo;ve set up a relatively simple Cloud Gateway, similar to what I described in my other post. On top of that I\u0026rsquo;ve created another WireGuard server which acts as a tunnel into my internal network for trusted user devices like my phone when I\u0026rsquo;m out of the house. Let me draw it out for you:\ngraph LR Phone(My Phone) Internet[Internet] Gateway(Cloud Gateway) CGNAT(ISP's CGNAT Router) subgraph House [House] Router(Router) Server(Server) Laptop(Laptop) end Internet --- CGNAT CGNAT --- Router Internet --- Gateway Phone -.-\u003e Gateway Phone --- Internet Gateway -.- CGNAT Gateway -.-\u003e CGNAT CGNAT -.- Router CGNAT -.-\u003e Router Router --\u003e Server Router --- Laptop So that\u0026rsquo;s terrible\u0026hellip; Oh well, I\u0026rsquo;ll try my best to explain it.\nFirstly, the solid lines indicate normal network connections. The dotted lines indicate a VPN connection.\nAs you can see, there\u0026rsquo;s one VPN line between My Phone and the Cloud Gateway. That\u0026rsquo;s the trusted VPN that terminates on the Router (which acts as a WireGuard \u0026ldquo;server\u0026rdquo;). This line (VPN) traverses the Cloud Gateway, and gets wrapped in another layer of WireGuard.\nThis second layer of WireGuard (the other dotted line) is the Peer to Peer VPN that allows us to have another WAN. In this case, the Cloud Gateway Server acts as the WireGuard \u0026ldquo;server\u0026rdquo; and the Router as the \u0026ldquo;client\u0026rdquo;. Because we are initiating the connection outward, we circumvent our inability to punch through the CGNAT router from the internet.\nFrom the perspective of the phone, it\u0026rsquo;s initiating a connection directly with the Cloud Gateway(\u0026rsquo;s IP). The Cloud Gateway just happens to be forwarding all its packets across another WireGuard connection to the Router. If you follow the arrows, you can see the path that the Phone\u0026rsquo;s VPN packets take to reach a web server on the Server.\nAt this point I can\u0026rsquo;t tell if this is extremely complicated or if I\u0026rsquo;ve just been overcomplicating it for the past 2 weeks. It could be that I\u0026rsquo;m just very used to it now 🤷‍♂️.\nImplementation # Ignoring the question of my sanity, let\u0026rsquo;s have a look at how I went about setting this up.\nThere are 3 entities involved in this dance of firewalls and VPNs that we\u0026rsquo;ll be interacting with:\nThe Phone The Gateway The Router To simplify the above diagram: The phone communicates with the gateway which, unbeknownst to the phone, forwards everything to the router.\ngraph LR Phone(My Phone) Gateway(Cloud Gateway) subgraph House [House] Router(Router) end Phone --\u003e Gateway Gateway --\u003e Router Between the Cloud Gateway and the Router, we\u0026rsquo;ll set up a VPN connection called the WAN VPN; named so because it tunnels traffic from the open internet without discretion (depending on how you configure it).\nBetween the Phone and the Router, we\u0026rsquo;ll set up a VPN connection called the Private VPN. This is because it grants access to the internal network of the router.\nLet\u0026rsquo;s go from least to most complex\u0026hellip;\nThe Phone # All we need to do for the phone is set it up as a WireGuard client on our Private VPN.\nIt does not need to know anything about the WAN VPN since, as I said before, the whole forwarding process is transparent.\nWe\u0026rsquo;ll configure the Phone as follows:\n[Interface] # Address of the client on the VPN Address = 192.168.17.69/24 # Wireguard client PrivateKey = ... [Peer] # Point to the gateway IP and the port that our private VPN \u0026#34;server\u0026#34; will listen on Endpoint = \u0026lt;gateway server public ip\u0026gt;:\u0026lt;13131\u0026gt; # Allow client to connect to VPN endpoint and internal network respectively AllowedIPs = 192.168.17.1/32, 192.168.16.0/24 # (Optional) Maintain the connection so we can initiate connections out to the device from the internal network PersistentKeepalive = 25 # Wireguard server PublicKey = ... The Gateway # Now we need the Cloud Gateway, acting as the middle man. It will act as a WireGuard server for our router to connect to.\nHere\u0026rsquo;s the configuration we\u0026rsquo;re going to want:\n[Interface] PrivateKey = ... Address = 10.0.18.1/30 ListenPort = 51820 # Allow ip forwarding PreUp = sysctl net.ipv4.ip_forward=1 # Immediately accept connections on 2222 so we can still SSH into the cloud gateway PreUp = iptables -t nat -A PREROUTING -i enp0s6 -p tcp --dport 2222 -j ACCEPT PostDown = iptables -t nat -D PREROUTING -i enp0s6 -p tcp --dport 2222 -j ACCEPT # Immediately accept UDP connections on 51820 (the wireguard listen port above) PreUp = iptables -t nat -A PREROUTING -i enp0s6 -p udp --dport 51820 -j ACCEPT PostDown = iptables -t nat -D PREROUTING -i enp0s6 -p udp --dport 51820 -j ACCEPT # Destination NAT all other traffic to our router at 10.0.18.2 PreUp = iptables -t nat -A PREROUTING -i enp0s6 -j DNAT --to-destination 10.0.18.2 PostDown = iptables -t nat -D PREROUTING -i enp0s6 -j DNAT --to-destination 10.0.18.2 [Peer] PublicKey = .... AllowedIPs = 10.0.18.2/32 This is all the same as I\u0026rsquo;ve explained in other posts, but the gist is that we\u0026rsquo;re setting up a simple WireGuard server and adding a few firewall rules that accept SSH and WireGuard traffic and forward everything else to the connected \u0026ldquo;client\u0026rdquo;.\nThe Router # Now we\u0026rsquo;re bordering rocket science\u0026hellip; ok maybe not that crazy but still.\nI actually implemented this on a Mikrotik Router, so there\u0026rsquo;s no .conf file per se, but I\u0026rsquo;ve written out the equivalent for you here:\nInterface 0 (Gateway - Router)\n[Interface] PrivateKey = ... Address = 10.0.18.2/30 # Set up routes on table 100, not main Table=100 # Mark new connections started from wg0 PreUp = iptables -t mangle -A PREROUTING -i wg0 -m state --state NEW -j CONNMARK --set-mark 10 PostDown = iptables -t mangle -D PREROUTING -i wg0 -m state --state NEW -j CONNMARK --set-mark 10 # Mark packets not originating from wg0 that have the above conenction mark PreUp = iptables -t mangle -A PREROUTING ! -i wg0 -m connmark --mark 10 -j MARK --set-mark 10 PostDown = iptables -t mangle -D PREROUTING ! -i wg0 -m connmark --mark 10 -j MARK --set-mark 10 # Route packets with the above mark using table 100 PreUp = ip rule add fwmark 10 table 100 priority 456 PostDown = ip rule del fwmark 10 table 100 priority 456 # Route anything originating from the local wg interface using table 100 PreUp = ip rule add from 10.0.18.2 table 100 priority 456 PostDown = ip rule del from 10.0.18.2 table 100 priority 456 [Peer] PublicKey = ... AllowedIPs = 0.0.0.0/0 Endpoint = \u0026lt;cloud gateway public ip\u0026gt;:51820 PersistentKeepalive = 25 The above config connects to the cloud gateway and uses the conditional routing we went over in the original post. One thing to note is the AllowedIPs field being set to 0.0.0.0/0 (everywhere); This is needed because when the gateway forwards packets over the WireGuard connection, it doesn\u0026rsquo;t masquerade their source addresses, so they appear to come directly from their origin and hence, we need to reply to that origin.\nInterface 1 (Phone - Router)\n[Interface] PrivateKey = ... Address = 10.0.17.1/24 ListenPort = 13131 [Peer] PublicKey = ... AllowedIPs = 192.168.17.0/24 The above config acts as the internal user WireGuard server. Nothing special at all. This is what a phone/laptop would connect to when outside the home network in order to gain access to it.\nSending Traffic To Services # Great, so everything\u0026rsquo;s wired up. The only thing left to do is actually allow traffic to get to the actual services.\nThere are 2 types of services for our scenario. Either the service is on the router, e.g. A ssh server or web interface, or it\u0026rsquo;s on another host e.g. A Minecraft server.\nFor my purposes, I wanted to access the web interface of the router over the VPN. All I had to do was set the subnet as LAN in my Mikrotik server and the default firewalls took care of the rest.\nFor the Minecraft server, I needed to add a Destination NAT rule in the firewall that points any traffic coming in on 25565 to my brother\u0026rsquo;s laptop.\nI\u0026rsquo;ll also likely be doing the same in the near future for all the services that are running on my home server such as this blog and my mail server.\nThe Meat # This shouldn\u0026rsquo;t have taken me so long to figure out and as usual, the solutions I came up with seem trivial now. There are however a few learning points that came out of this process.\nMore specifically, I\u0026rsquo;ve learned one or two things about the inner workings of iptables/nftables. Firstly, for context, I\u0026rsquo;ve drawn out the table and chain flow:\ngraph TD PacketIn(Inbound Packet) PacketOut(Outbound Packet) subgraph PREROUTING PreRaw(Raw) ConnTrack1[[State/Connection Tracking]] PreMangle(Mangle) IsLocalSource{{localhost source?}} PreNat(NAT) end RD1[[Routing Decision]] IsForThisHost{{for this host?}} subgraph INPUT InMangle(Mangle) InFilter(Filter) InNat(NAT) end subgraph FORWARD ForMangle(Mangle) ForFilter(Filter) end RD2[[Routing Decision]] subgraph OUTPUT OutRaw(Raw) ConnTrack2[[State/Connection Tracking]] OutMangle(Mangle) OutNat(NAT) RD3[[Routing Decision]] OutFilter(Filter) end ReleaseOutbound[[Release to Outbound Interface]] subgraph POSTROUTING PostMangle(Mangle) IsLocalDest{{localhost dest?}} PostNat(NAT) end Process{Local Process} PacketIn --\u003e PreRaw PreRaw --\u003e ConnTrack1 ConnTrack1 --\u003e PreMangle PreMangle --\u003e IsLocalSource IsForThisHost --\u003e|Yes| InMangle IsForThisHost --\u003e|No| ForMangle PreNat --\u003e RD1 RD1 --\u003e IsForThisHost IsLocalSource --\u003e|No| PreNat IsLocalSource --\u003e|Yes| InMangle ForMangle --\u003e ForFilter ForFilter --\u003e ReleaseOutbound ReleaseOutbound --\u003e PostMangle PostMangle --\u003e IsLocalDest IsLocalDest --\u003e|Yes| PacketOut IsLocalDest --\u003e|No| PostNat InMangle --\u003e InFilter InFilter --\u003e InNat InNat --\u003e Process Process --\u003e RD2 RD2 --\u003e OutRaw OutRaw --\u003e ConnTrack2 ConnTrack2 --\u003e OutMangle OutMangle --\u003e OutNat OutNat --\u003e RD3 RD3 --\u003e OutFilter OutFilter --\u003e ReleaseOutbound PostNat --\u003e PacketOut I\u0026rsquo;ve adapted this diagram from Phil Hagan\u0026rsquo;s Version.\nThe Learnings # The NAT table is only traversed for STATEFUL connections.\nIf you\u0026rsquo;re trying to test connectivity with ICMP (Ping) packets, it won\u0026rsquo;t work because they\u0026rsquo;re completely stateless.\nWireGuard packets (sort of) go through the entire chain twice.\nWhen an encrypted WireGuard packet arrives, it goes through prerouting and then input. After that, the local WireGuard process unwraps the encryption and spawns the contained packet on the WireGuard interface. That packet then traverses the whole firewall again as a completely separate entity.\nMoral Of The Story # All these shenanigans could have been easily avoided if the rest of the internet actually supported IPv6. My brother\u0026rsquo;s friend\u0026rsquo;s ISP simply doesn\u0026rsquo;t have it. And Vodafone\u0026rsquo;s Cellular Network doesn\u0026rsquo;t support it either.\nWhile I was writing this post, I remembered that there\u0026rsquo;s quite a few transition mechanisms which allow cross-communication between IPv4 and IPv6 networks in different capacities.\nIf there exists a mechanism whereby I can have one VPS translate IPv4 packets to IPv6 packets and forward them on to my internal network, that might be the way to go. That way I may not need any crazy VPNs. The only problem left to solve would be the complete ambiguity as to whether or not the IPv6 space that my ISP provides me is actually static\u0026hellip; A problem for another day.\nFor now, ciao 🫡.\n","date":"20 July 2024","externalUrl":null,"permalink":"/posts/recursive-vpns/","section":"Posts","summary":"VPNs inside VPNs inside VPNs inside VPNs.","title":"Recursive Wireguard","type":"posts"},{"content":"","date":"20 July 2024","externalUrl":null,"permalink":"/tags/vpn/","section":"Tags","summary":"","title":"Vpn","type":"tags"},{"content":" Everyone Wants Access to My Server # My server is accessible from the public internet, it has to be for people on the internet to read this blog! That\u0026rsquo;s all well and good, but I also made the decision to keep SSH open to the internet on port 22 as well.\nNow don\u0026rsquo;t worry, my config is tighter than a nun\u0026rsquo;s you-know-what. I have it set to disable password auth and to utilise an AllowList of users. Nobody except me is getting into this thing, assuming I don\u0026rsquo;t leak my keys one day 😬.\nThat\u0026rsquo;s not a challenge by the way, please don\u0026rsquo;t hack me 👉👈.\nRegardless, looking at the logs shows numerous bad actors constantly trying to get their grubby little fingers into my server:\nJun 28 22:16:10 sshd[811268]: User root from 68.183.91.213 not allowed because not listed in AllowUsers Jun 28 22:16:11 sshd[811268]: Connection closed by invalid user root 68.183.91.213 port 58890 [preauth] Jun 28 22:16:51 sshd[811908]: Invalid user admin from 174.138.57.202 port 45720 Jun 28 22:16:51 sshd[811908]: Connection closed by invalid user admin 174.138.57.202 port 45720 [preauth] Jun 28 22:19:25 sshd[814066]: User root from 68.183.91.213 not allowed because not listed in AllowUsers Jun 28 22:19:26 sshd[814066]: Connection closed by invalid user root 68.183.91.213 port 40382 [preauth] Jun 28 22:21:01 sshd[815334]: error: kex_exchange_identification: read: Connection reset by peer Jun 28 22:21:01 sshd[815334]: Connection reset by 139.59.1.230 port 37034 Jun 28 22:22:45 sshd[816640]: User root from 68.183.91.213 not allowed because not listed in AllowUsers Jun 28 22:22:45 sshd[816640]: Connection closed by invalid user root 68.183.91.213 port 36576 [preauth] Jun 28 22:23:20 sshd[817161]: User root from 174.138.57.202 not allowed because not listed in AllowUsers Jun 28 22:23:20 sshd[817161]: Connection closed by invalid user root 174.138.57.202 port 52472 [preauth] Jun 28 22:26:04 sshd[819289]: User root from 68.183.91.213 not allowed because not listed in AllowUsers Jun 28 22:26:05 sshd[819289]: Connection closed by invalid user root 68.183.91.213 port 55956 [preauth] Jun 28 22:29:25 sshd[822046]: User root from 68.183.91.213 not allowed because not listed in AllowUsers Jun 28 22:29:25 sshd[822046]: Connection closed by invalid user root 68.183.91.213 port 46956 [preauth] Jun 28 22:29:50 sshd[822336]: Invalid user ossuser from 174.138.57.202 port 44322 Jun 28 22:29:50 sshd[822336]: Connection closed by invalid user ossuser 174.138.57.202 port 44322 [preauth] Jun 28 22:36:19 sshd[828809]: Invalid user admin from 174.138.57.202 port 42276 Jun 28 22:36:19 sshd[828809]: Connection closed by invalid user admin 174.138.57.202 port 42276 [preauth] Jun 28 22:37:48 sshd[830252]: banner exchange: Connection from 47.95.215.141 port 50584: invalid format Jun 28 22:39:59 sshd[830546]: fatal: Timeout before authentication for 47.95.215.141 port 55398 Jun 28 22:41:18 sshd[831937]: fatal: Timeout before authentication for 47.95.215.141 port 40214 Jun 28 22:42:02 sshd[832636]: fatal: Timeout before authentication for 47.95.215.141 port 42336 Jun 28 22:42:48 sshd[835468]: Invalid user f5 from 174.138.57.202 port 60250 Jun 28 22:42:48 sshd[835468]: Connection closed by invalid user f5 174.138.57.202 port 60250 [preauth] Jun 28 22:45:51 sshd[838779]: Connection closed by 91.92.251.164 port 10000 Jun 28 22:49:18 sshd[841414]: Invalid user f5user from 174.138.57.202 port 56700 Jun 28 22:49:18 sshd[841414]: Connection closed by invalid user f5user 174.138.57.202 port 56700 [preauth] Most appear to just try their luck with the root account, some even try multiple times; I\u0026rsquo;m looking at you, Mr 68.183.91.213. It\u0026rsquo;s like he thinks my config is just going to magically change one day and let him in 😂.\nAnother common attack is to just try a bunch of different usernames to see if anything sticks. For example, Mr 174.138.57.202 really wants my server to be an F5 Firewall Device.\nThen there are some logs that I don\u0026rsquo;t understand, like the fatal: Timeout before authentication lines. My guess is that there\u0026rsquo;s a CVE in a particular version of SSHD (not mine) that 47.95.215.141 wants to exploit.\nWhat Can I Do About It? # Well option number 1 is nothing\u0026hellip; After all, they\u0026rsquo;re not getting in and due to the configuration, never will.\nBut that just isn\u0026rsquo;t satisfying to me, I want to slam the ban hammer and stop them from ever trying again.\nI had a look through some options including Fail2Ban but ultimately settled on Crowdsec because of its main selling point.\nWhat\u0026rsquo;s So Special About Crowdsec? # Crowdsec is based around the premise of crowd-sourced security.\nSay for example server A starts getting attacked by some IP Q and blocks it. Server B, owned by a completely different person, in a different country, would then be told to block Q because it is malicious.\nObviously all of this is anonymized so nobody knows who got attacked, only that the attack happened and who the perpetrator was (alongside some extra non-identifiable metadata).\nAnyway, I like this idea. It means that if someone gets attacked by an SSH brute-force, I don\u0026rsquo;t have to worry about the attacker turning their sights on me because I\u0026rsquo;ll already be blocking them.\nSetting Up # Crowdsec is relatively simple to get started with, there are 2 main components:\nThe crowdsec package itself. It comes with a systemd service that should be enabled and started.\nA crowdsec remediation component or bouncer to enact any suggestions that come out of the crowdsec service.\nCrowdsec Package # The crowdsec package is actually relatively useless right out of the box. We haven\u0026rsquo;t told it where to look or what to do if it finds something. Let\u0026rsquo;s start with the first problem.\nLog Acquisition # We can tell crowdsec where to look by configuring the /etc/crowdsec/acquis.yaml file (short for acquisition). You may not need to do this if all the log files you want to look at are already listed in the file. However, I did have to make some changes. My server runs ArchLinux which doesn\u0026rsquo;t put SSHD logs in a file, it instead uses the systemd journal. I had to add the following to get crowdsec to look for sshd logs in journald:\n... --- source: journalctl journalctl_filter: - \u0026#34;_SYSTEMD_UNIT=sshd.service\u0026#34; labels: type: syslog If you\u0026rsquo;re following along, you\u0026rsquo;ll want to do a quick systemctl reload crowdsec.service to get it to reload the config and start searching the right place. To check that this is working, we\u0026rsquo;ll look at the crowdsec logs using tail /var/log/crowdsec.log. You\u0026rsquo;re looking for the line that looks like:\ntime=\u0026#34;2024-06-28T22:56:54+01:00\u0026#34; level=info msg=\u0026#34;Running journalctl command: /usr/bin/journalctl [journalctl --follow -n 0 _SYSTEMD_UNIT=sshd.service]\u0026#34; src=\u0026#34;journalctl-_SYSTEMD_UNIT=sshd.service\u0026#34; type=journalctl If you see it, crowdsec is reading your SSHD logs!\nLog Parsing # Now that crowdsec knows where the logs are, we need to tell it what it should be looking for within them. That\u0026rsquo;s where parsers come in.\nA parser essentially consists of a set of regex rules and output configs. They categorize each line for the next stage of the process.\nAdding a parser is very easy:\ncscli parsers install crowdsecurity/whitelists This will add the whitelist parser, it essentially drops all events whose source is within internal network CIDRs (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12), that is assuming you trust them.\nNext we\u0026rsquo;ll add one to actually read the SSHD logs that we set up earlier:\ncscli parsers install crowdsecurity/sshd-logs This will add the official sshd-logs parser from the crowdsec hub. It specifically targets sshd logs which it will get from journald like we configured earlier.\nNow we systemctl reload crowdsec.service again and check to see if it\u0026rsquo;s working. For this we\u0026rsquo;ll run a:\ncscli parsers inspect crowdsecurity/sshd-logs If under Current metrics:, you don\u0026rsquo;t see anything, don\u0026rsquo;t worry. Log out of your SSH session and back in then re-run the command above. You should see 1 or more hits now.\nScenarios # So we\u0026rsquo;ve seen how we get the logs and how we read them, now what? Well we want to decide what to do about the information we now have. Enter Scenarios.\nScenarios are the meat of Crowdsec. They ingest events from the parsing stage and then, using a configurable set of rules, they tell us when to ban a given IP.\nWe\u0026rsquo;ll add 2 scenarios that both handle sshd events and determine whether an IP is performing a brute-force attack:\ncscli scenarios install crowdsecurity/ssh-bf cscli scenarios install crowdsecurity/ssh-slow-bf Once again, don\u0026rsquo;t forget to reload the crowdsec service.\nNow let\u0026rsquo;s test these scenarios, shall we?\nTesting # If you\u0026rsquo;ve already installed a bouncer, you will likely ban yourself. You\u0026rsquo;ve been warned.\nThis testing won\u0026rsquo;t work if you\u0026rsquo;re connecting from a trusted CIDR as we discussed before.\nIn one SSH session, start tailing the crowdsec log:\ntail -f /var/log/crowdsec.log In another terminal, you\u0026rsquo;re going to try to brute force your own server:\nssh root@myawseomserver ssh root@myawseomserver ssh root@myawseomserver ssh root@myawseomserver ssh root@myawseomserver Now if you check your logs, you\u0026rsquo;ll see that crowdsec wants to ban you!\ntime=\u0026#34;2024-06-29T09:55:19+01:00\u0026#34; level=info msg=\u0026#34;(6ae4ae5d27c646438d9444fa526b0a41/crowdsec) ssh-bf by ip 14.29.197.54 (CN/4134) : 4h ban on Ip 14.29.197.54\u0026#34; Here\u0026rsquo;s Mr 14.29.197.54 getting banned on my server a little while ago.\nNow before we install a bouncer (that will actually ban you), let\u0026rsquo;s undo that decision:\ncscli decision list Grab the id of that decision and then stuff it into:\ncscli decision delete --id \u0026lt;id\u0026gt; And just to make sure you\u0026rsquo;re not going to be banned as soon as we install the bouncer, this should return nothing:\ncscli decision list Bouncer (Remediation Component) # Crowdsec is now making decisions on who to block and for how long but as we\u0026rsquo;ve seen in testing, it doesn\u0026rsquo;t actually block anything. This is because crowdsec is missing a remediation component; Something to do the blocking.\nSince we\u0026rsquo;re looking to secure SSH, we\u0026rsquo;ll want to get crowdsec to block IPs in the firewall on our machine. This is where the cs-firewall-bouncer comes in. It makes itself at home in either your nftables or iptables firewall and links out to the crowdsec service to receive block lists.\nYou\u0026rsquo;ll want to install it via your package manager as it\u0026rsquo;s not a part of the crowdsec service itself.\n# ArchLinux AUR yay -Syu cs-firewall-bouncer Since it\u0026rsquo;s on the same machine, it should auto-connect to the crowdsec service.\nNow you have a working crowdsec installation. It will block anyone trying to brute-force their way into your server via SSH!\nI Want A Bigger Hammer # What we\u0026rsquo;ve just installed works great and will stop most attacks. However, I\u0026rsquo;m not satisfied. The 2 scenarios we installed are still quite lenient, and so I still see a lot of attempts, the most common of which are root login attempts.\nLogging in as root is disabled, so I can safely assume that anyone trying to log in as root is a bad actor.\nSo I created my own scenario /etc/crowdsec/scenarios/ssh-root.yaml:\n# ssh root attempt type: trigger name: ssh-root description: \u0026#34;Detect root login attempt\u0026#34; filter: \u0026#34;evt.Meta.log_type == \u0026#39;ssh_failed-auth\u0026#39; \u0026amp;\u0026amp; evt.Parsed.sshd_invalid_user == \u0026#39;root\u0026#39;\u0026#34; groupby: evt.Meta.source_ip blackhole: 1m reprocess: true labels: service: ssh confidence: 3 spoofable: 0 label: \u0026#34;SSH Root Attempt\u0026#34; behavior: \u0026#34;ssh:bruteforce\u0026#34; remediation: true Take a look at the filter, it specifies events that have failed ssh auth where the user is also root. The type is trigger, so the action occurs as soon as this scenario is triggered.\nSo this new scenario will immediately ban anyone who attempts to login as root. We have our bigger hammer.\nWhere\u0026rsquo;s The Crowd-Sourced Block-list? # Having run through all this, we haven\u0026rsquo;t actually touched on the main point of crowdsec, the community block-list.\nThat\u0026rsquo;s because we don\u0026rsquo;t have to! The crowdsec service automatically updates the community block-list in the background and applies it to the bouncer for you.\nYou can confirm this by counting the lines in the list:\nipset list crowdsec-blacklists | wc -l The reason we set up all the above is so that we can block live threats ourselves. The blocklist update frequency depends on how much we contribute. So if we parse and block locally, we\u0026rsquo;ll get faster updates and more up-to-date security.\nConclusion # In this post, we\u0026rsquo;ve learned how to install and configure crowdsec on a server to protect it from SSH brute-force attacks using crowd sourced blocklists and live log reading.\nThere\u0026rsquo;s still a lot to learn around crowdsec, I\u0026rsquo;ve barely scratched the surface with this post. Hopefully you found it useful; I look forward to seeing your contributions to the community block-list soon 😉.\n","date":"29 June 2024","externalUrl":null,"permalink":"/posts/crowdsec/","section":"Posts","summary":"A short foray into the world of Crowdsec.","title":"Crowdsec","type":"posts"},{"content":"","date":"29 June 2024","externalUrl":null,"permalink":"/tags/security/","section":"Tags","summary":"","title":"Security","type":"tags"},{"content":"","date":"21 June 2024","externalUrl":null,"permalink":"/tags/docker/","section":"Tags","summary":"","title":"Docker","type":"tags"},{"content":"","date":"21 June 2024","externalUrl":null,"permalink":"/tags/golang/","section":"Tags","summary":"","title":"Golang","type":"tags"},{"content":"","date":"21 June 2024","externalUrl":null,"permalink":"/tags/opensource/","section":"Tags","summary":"","title":"Opensource","type":"tags"},{"content":"","date":"21 June 2024","externalUrl":null,"permalink":"/tags/spotify/","section":"Tags","summary":"","title":"Spotify","type":"tags"},{"content":" All I Want To Do Is Share My Songs # I have a friend whose musical taste is a real mixed-bag. He\u0026rsquo;s simultaneously stuck in the 2000s pop era and somehow also has an absolute blast listening to my music. On numerous occasions he has asked me to share my playlist with him, but alas I\u0026rsquo;ve had to remind him each time that it\u0026rsquo;s impossible.\nAre you trying to share your \u0026ldquo;Liked Songs\u0026rdquo; now? Don\u0026rsquo;t worry, I\u0026rsquo;ll explain what\u0026rsquo;s going on.\nYour \u0026ldquo;Liked Songs\u0026rdquo; may appear like they\u0026rsquo;re in a regular Spotify playlist, but as it turns out, it\u0026rsquo;s completely faked. Spotify managed to create a whole different concept with half the features of a regular playlist. You can\u0026rsquo;t share, or start a Jam using your Liked Songs \u0026ldquo;playlist\u0026rdquo; 🙄.\nMy friend isn\u0026rsquo;t an isolated instance, many people I know have also tried sharing their Liked Songs in vain. One resorted to manually adding all their liked songs to a separate playlist and never using the \u0026ldquo;Like\u0026rdquo; button again\u0026hellip;\nI\u0026rsquo;m a Programmer\u0026hellip; I Can Fix This # Last weekend, I had a stroke of determination and slapped together a solution using the Spotify API. All it does is copy your Liked Songs to a new, regular playlist that you can actually share. After a brief foray with the name \u0026ldquo;Spotisync\u0026rdquo;, I realised that Spotify doesn\u0026rsquo;t like tools that start with \u0026ldquo;Spot\u0026rdquo; 🙄😂 and dubbed mine Syncify.\nIf I did my job right, you\u0026rsquo;re wondering what my music taste is\u0026hellip; And thanks to my tool, you can now see for yourself!\nAnd no, I didn\u0026rsquo;t copy them all manually, there\u0026rsquo;s over 1000 songs in that playlist. You think I have the time for that crap? That\u0026rsquo;s why I\u0026rsquo;m a programmer! 😎\nHow I Built It # I chose a relatively simple tech stack for a relatively simple project:\nGolang (with Chi \u0026amp; Gomponents) HTMX Tailwind These tools don\u0026rsquo;t have any fluff. Minimal enough to provide an efficient developer experience while enabling reliability and maintainability.\nI started by setting up authentication via Spotify. This was a breeze thanks to my OAuth2 experience, so much so that I did it manually. I didn\u0026rsquo;t even realise that the Spotify library I intended to use had authentication abstracted away for me already 🥴. I left it using my manual method in the end. It works, doesn\u0026rsquo;t it!\nThe next step was to get the songs syncing. This involved a lot of playing around with the Spotify API (which happens to be particularly well documented 👏). After a few hours, I had a rather janky system working. The process ended up as follows:\nCreate or find an existing Syncify playlist Truncate the playlist (songs get duplicated otherwise) Read the user\u0026rsquo;s Liked Songs Write them into the empty playlist There\u0026rsquo;s definitely a speed improvement to be made here; if I were to parallelize the truncation and reading parts, returning users who need their playlist truncated would, in theory, have a better experience.\nLastly came polishing. I went to v0.dev and generated a simple but well styled ui (which I converted to Gomponents), setup some feedback with HTMX and set it up to build to a Docker container.\nSince the Spotify API is kinda slow, I might switch the simple post request to a websocket so that I can return live updates to the UI in a progress bar instead of just a spinner.\nLinks # For those who want to see all the nitty-gritty details, the source code is freely available here.\nUnfortunately as of writing this post, Spotify has not gotten around to approving the tool for public use. When the time comes though, it\u0026rsquo;s hosted and ready to go: Syncify.\nIf you really want to use the tool now:\nYou can follow the instructions on the GitHub page to host it yourself. Or you can get in touch with me and I can add you to the testers list. UPDATE: It has been approved! syncify.yottapanda.com\nConclusion # I doubt it would take Spotify long to fix this themselves, but they\u0026rsquo;re not doing it so\u0026hellip;\nHopefully this helps you music lovers out there. Enjoy 🫡.\n","date":"21 June 2024","externalUrl":null,"permalink":"/posts/syncify/","section":"Posts","summary":"Making a tool that finally lets you share your Spotify Liked Songs.","title":"Syncify","type":"posts"},{"content":" File What Now? # FileDepot is a tool along the lines of Nextcloud or Dropbox, which I alluded to in my last blog post. I\u0026rsquo;ve been working on it since roughly mid-March, and it\u0026rsquo;s coming along well if I do say so myself 🥳.\nIf we were face-to-face right now, you\u0026rsquo;d likely be asking me why on earth I\u0026rsquo;d re-invent the wheel. For that reason, I\u0026rsquo;ll answer it now; FileDepot is designed to be more modern, secure, reliable, integrable, open, and faster than the alternatives.\nFile Why Now? # I\u0026rsquo;ve been using Nextcloud for a few years now and while it \u0026ldquo;works\u0026rdquo;, it\u0026rsquo;s far from fast, or modern. Unfortunately its worst downfall is reliability. I\u0026rsquo;ve had numerous times when an update breaks one feature or another, and the Android app is solely an unfixable, nightmarish torrent of \u0026ldquo;upload conflict\u0026rdquo; notifications and somehow lacks the ability to bidirectionally sync a folder. And yes, I\u0026rsquo;ve spent numerous days trying to solve these problems.\nDropbox/Google Drive/etc. solve a lot of the problems I\u0026rsquo;ve seen with Nextcloud, but they\u0026rsquo;re all proprietary, non-self-hostable, and most don\u0026rsquo;t integrate with standards like WebDAV. Plus companies like Google just sell all your data 😇.\nIn the self-hostable category or under my general requirements, I\u0026rsquo;ve been hard-stuck to find anything that\u0026rsquo;s not Nextcloud. Hence, I said \u0026ldquo;screw it, I\u0026rsquo;m making my own\u0026rdquo;.\nFile How Now? # For now, Filedepot has 2 components itself and relies on a third (any OAuth2 provider with OIDC support).\nServer # The FileDepot server acts as the central storage location, it serves a WebDAV endpoint and a UI endpoint with further potential protocols in-mind for the future such as SFTP or SSH. All the authentication is currently done with OAuth2/OIDC and hence the server acts as both an OAuth2 Resource Server and Client (the WebDAV and UI endpoints respectively).\nIt\u0026rsquo;s currently entirely written in Golang, including the UI which is built using gomponents. I chose Golang for a couple of reasons: I wanted to learn the language, and it is literally built for making APIs from the syntax to the standard library.\nI\u0026rsquo;m contemplating scrapping the UI portion of the server and replacing it with a Jetpack Compose Multiplatform web build in order to reduce the overhead of 2 separate UI codebases. The server UI is barely an MVP at this point, so I\u0026rsquo;m perfectly fine with nuking it.\nAndroid App # I\u0026rsquo;m making an Android app first because it fits my use case and because, say it with me, ✨ Apple sucks ✨. I\u0026rsquo;m only half joking, of course; As I alluded to a second ago, I\u0026rsquo;ve built the Android app using Jetpack Compose which will allow me (or someone else) to cross-build for Web, Desktop and iOS down the line.\nAt it\u0026rsquo;s core, the app is simply a WebDAV client. However, its killer feature is the sync system I\u0026rsquo;ve designed and begun building into it. The sync system gives the user ultimate control over how files are transferred to and from a given device. Want to keep instantly upload any photos you take on your phone but retain the ability to delete them on-device without deleting them from the cloud? DONE. Want to bidirectionally sync your Obsidian vault? DONE. And the bit that makes me the happiest? No more \u0026ldquo;upload conflict\u0026rdquo; notifications 🥴, thanks to the conflict resolution rules.\nIdeally, I want the app to be compatible with any WebDAV server, not just the FileDepot one. This would allow those who are stuck on Nextcloud, for example, to switch to a superior device client before making the jump on the server side. I\u0026rsquo;ll endeavour to make a migration tool, but that\u0026rsquo;s a long way off at the moment.\nOther Clients # I know I\u0026rsquo;ve said it a couple of times but for those who are skimming this; Thanks to Jetpack Compose Multiplatform (JCM) I\u0026rsquo;ll be able to make clients for Android, iOS, Desktop and Web.\nThe alternatives to JCM this come down to Flutter or a separate codebase for each platform. I initially did start using Flutter but ran into a few issues, namely the lack of solid libraries for WebDAV or OIDC/OAuth2. Also, UI not being my strong suit, I opted for one codebase over multiple.\nNow, I\u0026rsquo;m aware that the conception is that the JVM is particularly slow and bulky compared to compiled languages like Go, Rust, etc. I would have loved to build the entire stack in Golang but alas, the support/community is just not sufficient to build something \u0026ldquo;real\u0026rdquo;.\nIf you\u0026rsquo;re yelling \u0026ldquo;React Native\u0026rdquo; at me right now, hear this: JavaScript was a mistake. Everything we\u0026rsquo;ve built on top of it is like trying to put makeup on a frog. Sure it helps, but you still could not pay me to kiss it.\nFile Where Now? # Hey, I can\u0026rsquo;t find the code! I thought you said this was Open Source?\nFor now, I\u0026rsquo;ve decided to keep this project closed source. At least until I\u0026rsquo;ve had the chance to get something that works and clean it up a bit. I can\u0026rsquo;t have you lot gawking at my garbage prototype code!\nI intend to, at minimum, open source the server component of the project. The clients are a different matter and I\u0026rsquo;m still deciding on what I want to do with them.\nI\u0026rsquo;d love to be able to make money out of this project:\nOne way would be to offer a managed solution where I run the server and store your data and I can do this while also making the server open source and freely available. Another would be to make the whole app or portions of it paid. Hence, my hesitation to open source the app. Advertising is something I\u0026rsquo;d rather steer clear of. If you have any experience finding a good monetization strategy for things like this, do get in touch! I am always happy to receive advice.\nRegardless, monetization isn\u0026rsquo;t important until I have a working solution, so that\u0026rsquo;s what I\u0026rsquo;m focusing on at the moment.\nFollowing Progress # This blog will be the best place to follow the progress of FileDepot for now. If enough people take an interest, I may create a mailing list or Discord Server for more regular updates.\nUntil the next update/blog post, 🫡.\n","date":"14 June 2024","externalUrl":null,"permalink":"/posts/filedepot-intro/","section":"Posts","summary":"Introducing my side project, a new file sharing and backup system.","title":"Introducing FileDepot","type":"posts"},{"content":"","date":"14 June 2024","externalUrl":null,"permalink":"/tags/oauth2/","section":"Tags","summary":"","title":"Oauth2","type":"tags"},{"content":"","date":"14 June 2024","externalUrl":null,"permalink":"/tags/storage/","section":"Tags","summary":"","title":"Storage","type":"tags"},{"content":" Life Update: I\u0026rsquo;ve Been Busy # My short hiatus from blogging is over, but I haven\u0026rsquo;t been idle! I have something big in the works; sneak peeks are on the way 😉.\nRemember When I Made a \u0026ldquo;Cloud Router\u0026rdquo;? # So it turns out, my cloud router exists already under a different guise: a network gateway but in the ✨ cloud ✨. You learn something new every day!\nI guess I\u0026rsquo;ll change the name to Cloud Gateway instead!\nA Quick TL;DR of my Cloud Router Post # I couldn\u0026rsquo;t reliably reach my home network in order to serve things like email and websites due to CGNAT. I created a VPS in Oracle Cloud (because it was free) and then I routed all my server\u0026rsquo;s traffic across that instance using Wireguard VPN.\nThe Client \u0026amp; Problem # Not long after implementing the Cloud Gateway for my Self-Hosting server, I ended up needing to do something similar for a client at work.\nWe were making a mobile app for this client that showcased their products and allowed owners to manage their warranties and track servicing on the blockchain. The fancy-sounding stuff is irrelevant for this post though 😉.\nThe client has an existing CMS system (database 🙄) in which they store details about their physical products. Being a less technical company, they opted to host a FileMaker \u0026ldquo;database\u0026rdquo; on a Mac Mini in their office. Yeah, I\u0026rsquo;d never heard of it either.\nCalling the Mac directly from worldwide mobile devices via this app would likely melt the poor thing. Hence, in order to display all the products in the app, we needed to do an ETL (Extract, Transform, Load) from their on-premises database into a database in AWS.\nThis FileMaker database wasn\u0026rsquo;t on the public internet so instead when they needed remote access, they utilized a VPN system called ZeroTier which their employees used to connect to the database when out of the office.\nEnter the Cloud Gateway # I heard all this from the client and my thoughts went immediately to my Cloud Gateway implementation. It seemed the clear choice given the existing VPN. All I\u0026rsquo;d have to do was hook it up to an EC2 instance, do some network config and boom.\nAs it turns out, there\u0026rsquo;s a few gotchas when it comes to AWS networking. ZeroTier, though, was quite intuitive.\nHere\u0026rsquo;s a diagram which shows the desired network architecture:\ngraph LR subgraph AWS Lambda(Lambda) --\u003e EC2(EC2 Instance) end EC2 -- ZeroTier --\u003e Mac subgraph Client Mac(Mac Mini) end The AWS Side # The first thing I did was spin up an EC2 instance, no need for anything large at all; I decided on a t3.nano. After all, it\u0026rsquo;ll only be forwarding packets! Your home router probably has less than half the RAM and CPU of a t3.nano.\nI also used Amazon Linux just as a standard. Feel free to use whatever you want; just know that some steps will need to be tweaked.\nI also added an instance role with the AmazonSSMManagedInstanceCore policy to the instance so that I didn\u0026rsquo;t need to fumble with SSH keys (I\u0026rsquo;ll do another short post on this at some point).\nSecurity Group # Now it\u0026rsquo;s not mandatory but, it\u0026rsquo;s best practice to create a minimally permissive security group for the instance so that we only ever allow expected traffic in or out. This is what we\u0026rsquo;ll need for an instance with ZeroTier and using SSM (not SSH) to connect to it:\nInbound # Port Protocol Source Description 9993 UDP 0.0.0.0/0 ZeroTier Protocol 443 TCP 10.0.0.10/32 IP of Mac on ZeroTier (HTTPS) Outbound # Port Protocol Destination Description 9993 TCP 0.0.0.0/0 ZeroTier Protocol 9993 UDP 0.0.0.0/0 ZeroTier Protocol 80 TCP 0.0.0.0/0 ZeroTier Yum Repo (HTTP) 443 TCP 0.0.0.0/0 Amazon SSM Endpoints (HTTPS) If you\u0026rsquo;re using SSH instead of SSM, you can forego the 443 in outbound and instead put a 22 in inbound.\nSource/Destination Check # Now for the first gotcha, we want this instance to forward packets that are not bound for that instance itself right? Well AWS, has kindly assumed that you usually don\u0026rsquo;t want to do that (a very good idea from a security stand point) so we need to change a setting in the instance configuration to get this working:\nInstance \u0026gt; Actions \u0026gt; Networking \u0026gt; Change source/destination check \u0026gt; Stop should be checked ✅\nThis stops AWS filtering out traffic that\u0026rsquo;s not directly bound for that instance\u0026rsquo;s IP address. This lets us send packets to the instance that are destined, in this case, for the ZeroTier network.\nRoute Table Entry # Since we\u0026rsquo;re using a lambda to query the database, we need to let the lambda know that it can reach the Mac via the cloud gateway instance. We\u0026rsquo;re assuming the lambda is already being run from within the VPC (which is not the default).\nAs we saw above, the Mac\u0026rsquo;s IP in the ZeroTier network is 10.0.0.10, so we\u0026rsquo;ll add an entry in the VPC\u0026rsquo;s route table:\nDestination Target 10.0.0.10/24 i-blahblahblah When you add the route, select the instance. When it shows up in the route table view, it will show as an Elastic Network Interface (e.g. eni-blahblahblah). This is normal and if you go into the ENI, you\u0026rsquo;ll see that it is attached to your instance and, it is actually the thing giving your instance it\u0026rsquo;s IP address.\nConfiguring the Instance # Enable Packet Forwarding # Using the following commands, enable IP forwarding and reload sysctl:\necho \u0026#34;net.ipv4.ip_forward = 1\u0026#34; \u0026gt;\u0026gt; /etc/sysctl.conf sysctl --system Install and Enable nftables # nftables is the successor to iptables. It has not really caught on properly just yet but since this is a brand-new setup, we\u0026rsquo;ll use it.\ndnf update -y dnf install nftables -y systemctl enable nftables Configure nftables # Now that we have nftables, we can configure it to forward packets. We\u0026rsquo;ll start by creating a file /etc/nftables/zerotier.nft:\ntable ip nat { chain postrouting { type nat hook postrouting priority srcnat; policy accept; oifname \u0026#34;zt*\u0026#34; counter packets 0 bytes 0 masquerade } } The jist of this configuration is that it will masquerade any packets leaving via an interface starting with zt. This means that it will change the source IP of the packets to that of itself before sending and upon a response, it will swap the destination back to the original source IP.\nIt\u0026rsquo;s likely not a perfect configuration but, it gets the job done. If you can recommend any improvements, reach out 😉\nWe also have to enable this config file so, we need to edit /etc/sysconfig/nftables.conf. This file is the one that gets loaded by the systemd unit we enabled earlier.\nflush ruleset include \u0026#34;/etc/nftables/zerotier.nft\u0026#34; This does 2 things: Removes all the rules currently in play and then includes the zerotier.nft file we just created.\nNow we do a quick systemctl start nftables to load the file and, we\u0026rsquo;re onto the next step.\nZeroTier Install and Setup # Zerotier has an installer on this page, we want the steps under Linux (DEB/RPM).\nI\u0026rsquo;ve copy pasta\u0026rsquo;d it here for your convenience 😉\ncurl -s https://install.zerotier.com | sudo bash Run through that and then run the following:\nsystemctl enable --now zerotier-one zerotier-cli join \u0026lt;your-zerotier-network-id\u0026gt; Assuming you have a private ZeroTier network, you\u0026rsquo;ll want to now allow your instance access.\nTesting # Now you can run a sanity ping:\nping 10.0.0.10 And hopefully all is well!\nTesting from a Lambda is left as an exercise to the reader (AKA. I can\u0026rsquo;t be bothered to write about setting up the Lambda, sue me 😂)\n","date":"30 May 2024","externalUrl":null,"permalink":"/posts/wild-cloud-router/","section":"Posts","summary":"Implementing a cloud router for a client.","title":"Cloud Router in the Wild","type":"posts"},{"content":"","date":"11 February 2024","externalUrl":null,"permalink":"/tags/email/","section":"Tags","summary":"","title":"Email","type":"tags"},{"content":" OAuth2.0 # I self-host a multitude of applications, many of which require authentication. This usually entails as many usernames and passwords as there are apps. To solve this problem, I set up most of my applications to use OAuth2.0 for authentication. This means that I have one identity provider (in my case Authentik) that handles logging into all the apps; One password, one 2FA code, and by logging into one app, I\u0026rsquo;m automatically logged into all of them. Neat, right?\nThe Problem # Among the list of apps that I self-host is an email server.\nYes, it\u0026rsquo;s the most time consuming one of them all but I have learnt a lot by setting it up and it\u0026rsquo;s more free and flexible than most other solutions like Proton (which appears to be a great alternative from what I\u0026rsquo;ve heard).\nAnyway, self-hosted email simply never caught up with big tech email in terms of authentication. While the likes of Google and Microsoft moved to XOAuth (a variation of OAuth), they stuck with simple old passwords.\nNow this is not a big deal, passwords are secure when done right and they work with everything under the sun. However, when my dad inevitably forgets his password, I have to now change it for him in 2 places, not just one. Still not a big deal you say? I have OCD I don\u0026rsquo;t care, I have to fix it.\nFamous last words\u0026hellip;\nBut What About LDAP? # Docker Mailserver (DMS) already has solid support for LDAP but while LDAP would do the job, I don\u0026rsquo;t really want 2 different auth systems to deal with even if only one acts as a source of truth. Since OAuth2 is the newer protocol, I\u0026rsquo;m going to stick with that.\nHonestly, I\u0026rsquo;m probably wrong in hindsight, but when I decided to make the pull request, I didn\u0026rsquo;t want to bother learning and using LDAP.\nDovecot # DMS uses a program called Dovecot as it\u0026rsquo;s Mail Deliver Agent (MDA). It\u0026rsquo;s responsible for receiving email from Mail Transfer Agents (MTAs) like Postfix (that DMS also uses) and putting it in the relevant mailboxes for the users of the server and then serving those mailboxes to users via the IMAP or POP3 protocols.\nThe core of the problem lies in getting Dovecot to use an OAuth provider for it\u0026rsquo;s \u0026ldquo;userdb\u0026rdquo; and \u0026ldquo;passdb\u0026rdquo;. For our purposes, the userdb is a mechanism to look up a user\u0026rsquo;s/mailbox\u0026rsquo;s existence and ability to log in. Similarly, the passdb is used to check that a user\u0026rsquo;s credentials are valid and are authorized to access the service. This distinction will be important later, I\u0026rsquo;m not just telling you for fun 😉.\nThe Solution # If you want a less coherent view of what went down, here\u0026rsquo;s the GitHub PR. Otherwise, enjoy the ride\u0026hellip;\nInitial Implementation # So I started the PR quite strong, utilizing the knowledge of those before me who commented on the original issue. I setup DMS, Roundcube and Authentik like this:\nsequenceDiagram actor User User -\u003e\u003e Roundcube: Request login Roundcube -\u003e\u003e User: Redirect user User -\u003e\u003e Authentik: Log in Authentik -\u003e\u003e Roundcube: Return token Roundcube -\u003e\u003e Dovecot: Login with token Dovecot -\u003e\u003e Authentik: Validate Token Authentik -\u003e\u003e Dovecot: Respond with user info Dovecot -\u003e\u003e Roundcube: Respond with mailbox Roundcube -\u003e\u003e User: Display mailbox This is a standard OAuth flow, and worked quite well apart from 2 rather annoying issues:\nThere is no way to use an email client that doesn\u0026rsquo;t support generic OAuth. This is because I had disabled all other authentication mechanisms and so a password is simply no longer accepted.\nEmails sent to a valid user won\u0026rsquo;t arrive unless that user has previously logged into the system. This is because Dovecot/Postfix, I don\u0026rsquo;t actually know which, has no way of querying users from Authentik via OAuth2 and so only knows of your existence once you log in for the first time. This is a nuisance to put it mildly.\nTrials and Deprecations # In an attempt to solve problem 1, I tried OAuth1.0\u0026rsquo;s Password Grant feature. This allows an authentication client to take a username and password from the user and pass them along to the authentication server to verify. (Un)fortunately this got deprecated quite quickly as, sneaky clients can just steal your credentials and impersonate you.\nMildly Maiming 2 Birds With 1 Sad Excuse for a Pebble # Problem number 2 boils down to the Dovecot userdb (told you it would come back to haunt us). OAuth simply provides no way of querying users which is something we need.\nNow there is a way to solve this problem but it\u0026rsquo;s not pretty. It involves the API of whatever OAuth provider you\u0026rsquo;re using, more specifically, using it to query the users instead of the relying on the OAuth2 protocol to have all the answers (which it doesn\u0026rsquo;t).\nThe conclusion the maintainers and I came to was to limit the scope of the PR; Instead of a full blown Auth provider, add OAuth2 support on top of either of the file provider or LDAP provider. So you can either use a DMS specific password to log in or log in via something like Roundcube using your OAuth credentials.\nInstead of replacing the Dovecot userdb and passed, we simply add another passdb and Dovecot automatically tries both.\nI also added an integration test by mocking an OAuth provider using python to ensure everything worked as expected.\nAfter a lot of documentation updates and conversations regarding configuration, the changes were eventually merged in.\nFuture Plans # I\u0026rsquo;d still like to implement full OAuth2 support by adding userdb capabilities via API calls. Unfortunately it doesn\u0026rsquo;t make much sense until more clients support generic OAuth.\nPerhaps if I were to make an RFC to add user querying to OAuth\u0026hellip; Who am I kidding, the chances of that happening are slim to none 😂\n","date":"11 February 2024","externalUrl":null,"permalink":"/posts/dms-oauth2/","section":"Posts","summary":"My journey implementing OAuth2 for Docker Mailserver.","title":"OAuth2 for Docker Mailserver","type":"posts"},{"content":" Act 1: Finding a New ISP # Internet speeds in the UK tend to be rather abysmal for their asking price.\nI\u0026rsquo;m currently buying an internet connection from Plusnet who are charging me ~£20/m for a 36Mb/s down, 10Mb/s up connection (with landline). They also charged me a small, one time setup fee (a few quid) for a static IPv4 address which I\u0026rsquo;ve been happily using for a while now. This is all provided over my copper phone line (very old).\nSelf-hosting somehow survives these speeds just fine for the websites and such that I run, but I\u0026rsquo;ve been eyeing up a faster connection for some time now to improve my file transferring capabilities. Unfortunately there is a speed limit on copper wires of 80Mb/s down, 20Mb/s up and that\u0026rsquo;s in the best case scenario. So my dreams of Gigabit internet are going to require a bit more effort.\nLet\u0026rsquo;s aim for that \u0026ldquo;Gigabit internet\u0026rdquo; target and see what we can find (ignoring contract length and setup fees):\nISP Down/Up Mb/s Cost/Month Virgin Media 1130/104 £45 BT 900/110 £43 Plusnet 900/115 £50 Sky 900/100 £58 Community Fibre 920/920 £25 \u0026ldquo;Do you see the odd one out?\u0026rdquo; - Dora the Explorer.\nClearly Community Fibre woke up and chose violence. Not only do they have the second-highest average down speed, they also boast a symmetric upload speed. All for half the price of the other offerings.\n\u0026ldquo;Yeah that\u0026rsquo;s great but what\u0026rsquo;s the catch?\u0026rdquo; is what you\u0026rsquo;re probably asking right about now. I asked it too.\nCarrier-Grade Network Address Translation (CGNAT) # CGNAT is the ISP\u0026rsquo;s solution to dwindling IPv4 address space. Let\u0026rsquo;s break it down:\nNetwork Address Translation (NAT) is the protocol that allows one public IPv4 address to be used by multiple devices, each with it\u0026rsquo;s own private IPv4 address. Generally, an ISP will hand your router/gateway a public IPv4 address and your router will give each device that connects to it a private one. Then as outgoing packets pass through the router, return addresses are switched from private ones to the public one and vice versa for the responses. It\u0026rsquo;s all very clever, and I suggest learning more about it; Perhaps I\u0026rsquo;ll go into more detail in a future post.\nSo what makes it Carrier-Grade? At it\u0026rsquo;s core, CGNAT is just a second layer of NAT. Where your home router may use the usual private IPs such as 192.168.0.0/16 or 10.0.0.0/8, CGNAT has the 100.0.0.0/8 range assigned to it. So it looks something like this:\ngraph LR A(Internet) \u003c-- 0.0.0.0/0 --\u003e B B(CGNAT Router) \u003c-- 100.42.42.0/24 --\u003e C C(Your Router) \u003c-- 192.168.0.0/24 --\u003e D(Your Server) Rather convoluted if you ask me!\nNow the problem with this is that we don\u0026rsquo;t have control over the ISP router that\u0026rsquo;s performing CGNAT, so we have no way of port forwarding. That means no self-hosting 😢.\nAct 2: A Challenger Approaches # Since we\u0026rsquo;re saving so much on our internet connection, what\u0026rsquo;s stopping us from spending a little money on implementing an equally convoluted way of getting around CGNAT?\nIf we add our own NAT gateway (router) in a public cloud, it will have a public static IPv4 address, a public static IPv6 address, and likely a fast internet connection too. We can then set up a Wireguard tunnel from our local server to that cloud router. We then forward all our packets via the new Cloud Router at which point we have essentially given our server a public static IPv4 address again!\nConfused? Yeah me too, so I made a diagram:\ngraph LR A(Internet) \u003c-- 0.0.0.0/0 --\u003e B B(CGNAT Router) \u003c-- 100.42.42.0/24 --\u003e C C(Your Router) \u003c-- 192.168.0.0/24 --\u003e D(Your Server) A \u003c-- 0.0.0.0/0 --\u003e Q(Cloud Router) Q \u003c-- 10.69.69.0/24 --\u003e D Physically, the Wireguard tunnel (10.69.69.0/24) goes over our existing CGNAT connection but logically, the CGNAT connection is abstracted away and isn\u0026rsquo;t something we need to worry about just yet.\nFinding a VPS # We\u0026rsquo;ll need something to act as our Cloud Router, so I\u0026rsquo;ve compiled a list of some various VPS options below. Each assumes that savings plans (or equivalent) have been applied up to 1 year in duration, excluding any upfront payments. I\u0026rsquo;ve also gone for the smallest VPS from each provider as we barely need any raw power.\nCloud vCPU RAM (GiB) Disk (GB) Data Transfer (GB) Network Speed (Gb/s) Monthly Cost ($) Cost Calculator Link (or equivalent) AWS 2.0 0.5 8 25 \u0026lt; 5.0 8.83 Link (New Elastic IP charge considered) GCP 0.25 1.0 10 200 1.0 5.44 Link Azure 1.0 0.75 8 100 \u0026lt; 6.25 4.74 Link Oracle Cloud 4.0 (ARM) 24.0 200 10 000 4.0 0.00 Link Oracle Cloud 2.0 1.0 200 10 000 0.48 0.00 Link Linode 1.0 1.0 25 1000 1.0 5.00 Link I see 3 potential solutions here in my personal order of preference:\nOracle for having a ludicrous free tier including both ARM and x86 machines with 10TB of egress. Linode for their apparent lack of immediately obvious problems and their extra features (which I\u0026rsquo;ll explain soon). Azure for simply being the cheapest on the board. A potential worry here is hitting the 100GB free data transfer limit in one month. (Azure also gets no points for their calculator, it\u0026rsquo;s slow). AWS appears far too expensive for this use case. However, it wasn\u0026rsquo;t the compute, the transfer cost is what\u0026rsquo;s stinging us.\nAWS loses in this battle because they don\u0026rsquo;t offer some free transfer every month. Egress transfer costs are expensive across the board beyond free amounts so if you\u0026rsquo;re looking to do multiple TB per month, Oracle is your choice.\nGCP doesn\u0026rsquo;t make the cut due to their high vCPU price, perhaps they have extremely fast CPUs, but I\u0026rsquo;ve not dug into it so for now\u0026hellip; we\u0026rsquo;ll leave them be.\nLinode\u0026rsquo;s \u0026ldquo;Extra Features\u0026rdquo; # Since we\u0026rsquo;re making a Cloud Router, wouldn\u0026rsquo;t it be nice if we could do some IPv6 routing? Linode is the only provider (from what I can tell) that allows us to assign a prefix larger than a /64 to our VPS. This means we can have it run SLAAC with RA and assign downstream clients public static IPv6 addresses.\nIf instead of putting the Wireguard endpoint on our server, we could put it on our usual router, and it would almost act as just another WAN connection!\nAnother nicety of Linode is the easy setup of Reverse DNS (PTR) records for our v4 IP. They have this functionality built into their management console, unlike Oracle. This is important if you host an email server like I do.\nChoose Your Fighter # We\u0026rsquo;ll go with Oracle for now since:\nIt\u0026rsquo;s free. I don\u0026rsquo;t particularly need a public static IPv6 address for everything at the moment. They still do offer RDNS via a support request. A quick aside; Oracle Cloud\u0026rsquo;s free tier is actually so good that I feel obligated to urge you to forego this convoluted setup and just host your stuff in their cloud directly.\nAct 3: Implementation # I won\u0026rsquo;t be mad if you skipped to this, I\u0026rsquo;ve been known to waffle.\nLet\u0026rsquo;s take an inventory of what we\u0026rsquo;ll need:\nA Wireguard \u0026ldquo;server\u0026rdquo; on the VPS since it\u0026rsquo;s the only thing with a public IP. A set of iptables rules to forward traffic correctly on the VPS. A Wireguard \u0026ldquo;client\u0026rdquo; on our local server. A set of iptables rules to route traffic back over the Wireguard tunnel. VPS # Once you\u0026rsquo;ve signed up for a free Oracle Cloud account, use the search in the management console to find the following sections.\nVirtual Cloud Network (VCN) # Create a VCN with a private IPv4 CIDR of something like 10.0.16.0/24.\nAdd an Internet Gateway to your VCN.\nEdit the default route table in your VCN, adding a route rule pointing to your previously added internet gateway. It should have a destination CIDR block of 0.0.0.0/0 since we want to route internet-bound traffic over the internet gateway (i.e. it will act as our default route).\nEdit the default security list for your VCN so that it has the following entries:\nSource Protocol Destination Port Explanation 0.0.0.0/0 TCP 22 For SSH-ing into the cloud router before we reconfigure it 0.0.0.0/0 TCP 2222 For SSH-ing into the cloud router once we\u0026rsquo;ve reconfigured it 0.0.0.0/0 TCP 51820 For Wireguard If you were particularly paranoid, you could lock down the Wireguard one to only allow Source IPs from your ISP\u0026rsquo;s CGNAT CIDRs.\nThere are 2 ICMP rules, you can either ignore these or delete them and add your own that allows all ICMP traffic. This may come in handy for troubleshooting.\nCreate a subnet with the same IPv4 CIDR as the VCN (10.0.16.0/24), use the default route table, make it public, and use the default security list. Reserved Public IP # Reserve a public IP from Oracle\nInstance # Pick Ubuntu as your operating system (if you want to use an ARM instance, pick the aarch64 image).\nChange the shape to the specs you want, the minimums for any architecture are fine. I\u0026rsquo;ll pick an ARM based instance with 2 vCPUs and 4 GB of RAM.\nThis is what the Image and Shape section should look like if you\u0026rsquo;re following this article to the letter: Under \u0026ldquo;Primary VNIC information\u0026rdquo; ensure it\u0026rsquo;s using all the relevant entities from the VCN that we just configured. The \u0026ldquo;auto assign public IPv4 address\u0026rdquo; box should be checked.\nAdd your SSH public key.\nNow you can press create.\nOnce the instance has spun up, you can connect to it over ssh, something like ssh ubuntu@\u0026lt;Public IP\u0026gt;.\nThen run the following commands on the cloud router:\napt install \u0026amp;\u0026amp; apt upgrade -y # ignore the service restarts for now apt install -y wireguard # ignore the service restarts for now cp /etc/iptables/rules.v4 /etc/iptables/rules.v4.orig # take a copy of the original iptables rules just in case sed -i \u0026#34;/icmp-host-prohibited/d\u0026#34; /etc/iptables/rules.v4 # Remove the Oracle REJECT rules from INPUT and FORWARD iptables-restore \u0026lt; /etc/iptables/rules.v4 # load the updated iptables rules reboot Once the machine has rebooted, reconnect and run the following commands: cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig # take a copy of the original sshd_config just in case sed -i \u0026#34;s/#Port 22/Port 2222/\u0026#34; /etc/ssh/sshd_config # Change the port that sshd listens on to 2222 From now until the next step do NOT disconnect from the session until you have tested your ability to ssh back in from a NEW shell. If you can\u0026rsquo;t SSH into your cloud router after this, terminate it and go back to step 1.\nsystemctl reload sshd # reload the sshd config (now listens on 2222) Now in a new terminal, ssh into the machine using the new port:\nssh -p 2222 ubuntu@\u0026lt;Public IP\u0026gt; If this works, you\u0026rsquo;re safe, otherwise check/undo the previous changes and try again.\nNow we can get onto Wireguard. Let\u0026rsquo;s create some keys: wg genkey | tee server.priv wg pubkey \u0026lt; server.priv | tee server.pub wg genkey | tee client.priv wg pubkey \u0026lt; client.priv | tee client.pub Now we create /etc/wireguard/wg0.conf on the cloud router with the following contents: [Interface] Address = 10.0.17.1/30 # Any CIDR that doesn\u0026#39;t overlap with the VCN\u0026#39;s CIDR or anything in your home network ListenPort = 51820 PrivateKey = # contents of server.priv goes here # Allow the router to forward packets PreUp = sysctl net.ipv4.ip_forward=1 # Immediately accept connections on 2222 so we can still SSH into the cloud router PreUp = iptables -t nat -A PREROUTING -i ens3 -p tcp --dport 2222 -j ACCEPT PostDown = iptables -t nat -D PREROUTING -i ens3 -p tcp --dport 2222 -j ACCEPT # Destination NAT all other TCP traffic to our local server at 10.0.17.2 PreUp = iptables -t nat -A PREROUTING -i ens3 -p tcp -j DNAT --to-destination 10.0.17.2 PostDown = iptables -t nat -D PREROUTING -i ens3 -p tcp -j DNAT --to-destination 10.0.17.2 [Peer] PublicKey = # contents of client.pub goes here AllowedIPs = 10.0.17.2/32 You\u0026rsquo;ll need to replace the ens3 interface name; you can check yours by doing an ip link show and looking for the first one beginning with \u0026ldquo;e\u0026rdquo;. It may look something like enp0s12.\nStart the Wireguard server: chown root:root /etc/wireguard/wg0.conf chmod go= /etc/wireguard/wg0.conf systemctl enable --now wg-quick@wg0 Ensure everything worked nicely using:\nsystemctl status wg-quick@wg0 Local Server # Now we\u0026rsquo;ll setup something very similar on our local server, it should be a bit simpler since we have most of what we need now.\nSSH into your local server.\nInstall wireguard/wireguard-tools depending on your distribution.\nCreate the client Wireguard config /etc/wireguard/wg0.conf:\n[Interface] Address = 10.0.17.2/30 PrivateKey = # client.priv goes here (the one we generated on the cloud router) Table = 123 # Use a new routing table just for wireguard stuff # Mark all connections originating from the wireguard interface PreUp = iptables -t mangle -A PREROUTING -i wg0 -m state --state NEW -j CONNMARK --set-mark 1 PostDown = iptables -t mangle -D PREROUTING -i wg0 -m state --state NEW -j CONNMARK --set-mark 1 # Mark all packets originating anywhere except the wireguard interface that match the connection mark from above PreUp = iptables -t mangle -A PREROUTING ! -i wg0 -m connmark --mark 1 -j MARK --set-mark 1 PostDown = iptables -t mangle -D PREROUTING ! -i wg0 -m connmark --mark 1 -j MARK --set-mark 1 # Route all packets with the mark using the new table PreUp = ip rule add fwmark 1 table 123 priority 456 PostDown = ip rule del fwmark 1 table 123 priority 456 # Route any responses that originally came to the wireguard interface back over it PreUp = ip rule add from 10.0.17.2 table 123 priority 456 PostDown = ip rule del from 10.0.17.2 table 123 priority 456 [Peer] PublicKey = # server.pub goes here (the one we generated on the cloud router) AllowedIPs = 0.0.0.0/0 Endpoint = \u0026lt;Cloud Router Public IP\u0026gt;:51820 # Ensure the connection doesn\u0026#39;t die so we can always receive new connections from the cloud router PersistentKeepalive = 30 And start it: chown root:root /etc/wireguard/wg0.conf chmod go= /etc/wireguard/wg0.conf systemctl enable --now wg-quick@wg0 Again, check the status with: systemctl status wg-quick@wg0 Act 4: Testing # If everything has gone well, we should have a working cloud router now!\nSSH # The best way to check this now is to SSH into your local server using the public IP of the cloud router, so:\nssh localServerUser@\u0026lt;Cloud Router Public IP\u0026gt; Let\u0026rsquo;s go through what\u0026rsquo;s happening when we execute this command:\nPacket from the initiator (in this case you) comes into the cloud router from the internet. The packet gets checked by the iptables rules in the NAT table, and it matches the second rule (iptables -t nat -A PREROUTING -i ens3 -p tcp -j DNAT --to-destination 10.0.17.2). That same rule changes the destination address to that of the local server (over the wireguard connection). The cloud router forwards the packet to the local server. The local server receives the packet and marks the connection before handling the packet. The local server generates a response on the connection and begins to send it back out. The local server marks the packet since the connection was already marked. The local server sends the response packet back over the Wireguard connection to the cloud router. The cloud router receives the packet and forwards it out back over the internet to the initiator (you). In this case, the connection/packet marking isn\u0026rsquo;t needed as the response packet\u0026rsquo;s from address is that of the Wireguard interface, so we just use that information to route it correctly.\nThe marking comes into play when you use (for example) docker to run the actual service\u0026hellip; a web server or something.\nWeb Server # Speaking of web servers, let\u0026rsquo;s set one up:\nAdd a rule in the Security List in your Oracle VCN for port 80 from 0.0.0.0/0.\nInstall docker on your local server and run the following command to start a basic web server:\ndocker run --rm -it -p 80:80 nginx Using a browser, go to http://\u0026lt;Cloud Router Public IP\u0026gt; and you should see the default Nginx landing page. A similar process can be employed to expose whatever service you want to.\nAct 5: Conclusion # Hopefully this has worked out alright for you and helped you circumvent your CGNAT (or equivalent) situation. It\u0026rsquo;s working beautifully for me so far. If you have questions, I might have a social link on the home page.\nI wouldn\u0026rsquo;t have been able to write this article without the help of Justin Ludwig and his 2 articles WireGuard Port Forwarding From the Internet and WireGuard Port Forwarding From the Internet to Other Networks. His graphics are much cooler than mine, so if something doesn\u0026rsquo;t make sense here, his will.\n","date":"13 January 2024","externalUrl":null,"permalink":"/posts/cloud-router/","section":"Posts","summary":"Getting around CGNAT for self-hosting with a Cloud Router.","title":"Cloud Router","type":"posts"},{"content":"","date":"22 December 2023","externalUrl":null,"permalink":"/tags/hugo/","section":"Tags","summary":"","title":"Hugo","type":"tags"},{"content":"","date":"22 December 2023","externalUrl":null,"permalink":"/tags/nginx/","section":"Tags","summary":"","title":"Nginx","type":"tags"},{"content":" Introduction # I\u0026rsquo;ve had a website for a few years now. The first iteration I made was a simple, one page HTML site with just a few links. That evolved to also show off a few projects of mine before devolving to a SvelteKit backend with just a few links again. This was always deployed as a docker container on my home server (which has amounted to a lot of headaches, but that\u0026rsquo;s for another post).\nRecently I decided it was time to actually get some work done, make this blog, revamp the site and build my \u0026ldquo;personal brand\u0026rdquo; or whatever you want to call it. This post details just the start of the journey to make this website a fountain of knowledge and something to be proud of.\nDecisions, Decisions # As it turns out, there\u0026rsquo;s a lot of ways to make a website nowadays (I\u0026rsquo;m not that old I promise). For any interactivity, you probably want to go for one of the major frameworks like React or if you like more niche solutions, Svelte(Kit) might be a good bet. Thankfully for this re-imagining of my website, all I need is a blog and a few links; A static site generator would do nicely, please and thank you.\nWhat\u0026rsquo;s a static site generator (SSG) you ask? Well they\u0026rsquo;re quite neat in my opinion. They allow you to take markdown documents which are just glorified text files and auto-magically format them into a webpage. (I say magic when I really just mean applying a predefined style and formatting guide).\nThere\u0026rsquo;s a few SSGs running the blogging side of the internet at the moment:\nNext.js Hugo Gatsby Jekyll MkDocs a lot more Each of the big players has a focus in which they excel, for example Next.js is great at combining the features of an SSG with a full featured framework, React in this case. Whereas something like Jekyll is designed more for static blogs.\nHugo caught my eye with it\u0026rsquo;s build process and structure. It makes it easy to setup and then focus on the content with simple templating and taxonomies (e.g. tags and categories). So that\u0026rsquo;s what I ended up with!\nTheming # Having a look through some themes, I decided to first try PaperMod since it:\nWas the first one on the list Looked good Had a lot of stars on GitHub Unfortunately this turned out to be a bit of a mistake; The installation, configuration and usage process of PaperMod\u0026hellip; Could use some work. I found that the documentation wasn\u0026rsquo;t quite up to scratch and I ended up trawling through examples from other people\u0026rsquo;s blogs to find and change what I needed to.\nAt some point in this struggle I decided to give up and try a different theme which is when I came across Congo. The docs are great, the examples are thorough and the configuration was easy. It even comes with a decent set of color schemes out of the box.\nBuilding # After making an initial page and running the hugo server command on my development machine, I found a great looking blog hosted locally.\nThe Hugo docs tell us to use the hugo command to build our site. This command produces a \u0026ldquo;public\u0026rdquo; folder which can be served by any webserver of your choosing (Nginx or Apache spring to mind).\nSince we want to deploy this site to a container environment (specifically docker-compose on my home server), we\u0026rsquo;ll need to build a container image. I\u0026rsquo;ve always used docker build to accomplish this and so that\u0026rsquo;s what we\u0026rsquo;ll do now. I started out with something like this:\nFROM klakegg/hugo AS build WORKDIR /build COPY . . RUN hugo FROM nginx:alpine COPY --from=build /build/public /usr/share/nginx/html/ The klakegg/hugo build image was one I found after searching around online for something popular, I failed to realize it hadn\u0026rsquo;t been updated in a while. While the hugo command worked locally (I use Arch btw), this docker build was not happy:\nERROR 2023/12/22 14:27:49 render of \u0026#34;taxonomy\u0026#34; failed: \u0026#34;/build/themes/congo/layouts/_default/baseof.html:5:12\u0026#34;: execute of template failed: template: _default/taxonomy.html:5:12: executing \u0026#34;_default/taxonomy.html\u0026#34; at \u0026lt;site\u0026gt;: can\u0026#39;t evaluate field LanguageCode in type *langs.Language ERROR 2023/12/22 14:27:49 render of \u0026#34;taxonomy\u0026#34; failed: \u0026#34;/build/themes/congo/layouts/_default/baseof.html:5:12\u0026#34;: execute of template failed: template: _default/taxonomy.html:5:12: executing \u0026#34;_default/taxonomy.html\u0026#34; at \u0026lt;site\u0026gt;: can\u0026#39;t evaluate field LanguageCode in type *langs.Language Total in 20 ms Error: Error building site: failed to render pages: render of \u0026#34;home\u0026#34; failed: \u0026#34;/build/themes/congo/layouts/_default/baseof.html:5:12\u0026#34;: execute of template failed: template: index.html:5:12: executing \u0026#34;index.html\u0026#34; at \u0026lt;site\u0026gt;: can\u0026#39;t evaluate field LanguageCode in type *langs.Language What on earth does all that mean!?\nThankfully we don\u0026rsquo;t have to care too much as we have a scenario where it\u0026rsquo;s building just fine, local. ArchLinux usually has particularly up-to-date packages and that holds true for the hugo package. It turns out that the hugo version built by klakegg a few months ago is already so outdated that the above errors are occurring.\nSo what\u0026rsquo;s the solution? Build the site on Arch, of course!\nFROM archlinux:latest AS build RUN pacman -Sy hugo --noconfirm WORKDIR /build COPY . . RUN hugo FROM nginx:alpine COPY --from=build /build/public /usr/share/nginx/html/ Now, while this works, this has it\u0026rsquo;s downsides:\n1. Downloading the hugo package every build isn\u0026rsquo;t exactly quick, nor efficient. # It would be much better to pre-build the build image and store it separately to use here. This means we only have to fetch the hugo package once in a while rather than every build. The problem with this is that it\u0026rsquo;s another repository to manage, and monitor. For something this simple, it\u0026rsquo;s probably just not worth the hassle. Especially with layer caching, the impact isn\u0026rsquo;t really noticeable for subsequent builds.\n2. ArchLinux is rolling release # This means that the hugo package can be updated whenever. This might introduce breaking changes which could cause the site to start failing to build with no warning. Ideally we\u0026rsquo;d want to pin to a specific version of Hugo, likely running it from source with Go at some commit ref or tag.\nSecurity # Now that we have the site building and running well, we should consider security. I once had a job interview in which my interviewer had looked up my old website and checked it\u0026rsquo;s security. He showed me a website that verified the existence and configuration of a few headers that I\u0026rsquo;d never even heard of before:\nContent-Security-Policy Permissions-Policy Referrer-Policy Strict-Transport-Security X-Content-Type-Options X-Frame-Options As it so happens, these headers are core to modern internet security, providing the ability to prevent various attacks such as Cross Site Scripting or ClickJacking and generally improving the safety of your site for your viewer. As an example, here\u0026rsquo;s the report for this website.\nTo set these up with Nginx, we simply have to add the headers to the configuration file (/etc/nginx/nginx.conf) under one of three sections:\nThe http section covers the entire Nginx server and all routes it serves. The server section covers just that one server. The location sections covers just that one path/subdomain. I decided to put mine in the http section since this Nginx instance will only ever serve this one site and I want everything to be covered. Here\u0026rsquo;s what my config file looks like:\n... http { ... add_header Strict-Transport-Security \u0026#39;max-age=31536000; preload\u0026#39;; add_header Content-Security-Policy \u0026#34;default-src \u0026#39;self\u0026#39;; font-src *;img-src * data:; script-src *; style-src * \u0026#39;unsafe-inline\u0026#39;\u0026#34;; add_header X-Frame-Options \u0026#34;SAMEORIGIN\u0026#34;; add_header Referrer-Policy \u0026#34;strict-origin\u0026#34;; add_header Permissions-Policy \u0026#34;geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(),payment=()\u0026#34;; add_header X-Content-Type-Options nosniff; server { listen 80; listen [::]:80; server_name localhost; location / { root /usr/share/nginx/html; } } } And setting this up in the Docker image we end up with our Dockerfile as such:\nFROM archlinux:latest AS build RUN pacman -Sy hugo --noconfirm WORKDIR /build COPY . . RUN hugo FROM nginx:alpine COPY target/etc/nginx/nginx.conf /etc/nginx/nginx.conf # Copy our new config file COPY --from=build /build/public /usr/share/nginx/html/ Running # By running the following command, we get our website running locally on localhost using Docker:\n# Assuming you\u0026#39;re in the directory with your Dockerfile docker build -t my-blog . \u0026amp;\u0026amp; docker run --rm -p 80:80 my-blog Now you can publish that image and deploy it anywhere you want, which I do in the linked posts (to be written).\n","date":"22 December 2023","externalUrl":null,"permalink":"/posts/website/","section":"Posts","summary":"The evolution of this website and how it’s built.","title":"This Website","type":"posts"},{"content":"","date":"22 December 2023","externalUrl":null,"permalink":"/tags/website/","section":"Tags","summary":"","title":"Website","type":"tags"},{"content":"So I recently managed to make my Self-Hosted mailserver an Open Relay. This is bad.\nMy mailserver (dockerized mailcow) currently runs on a little NUC under my stairs. It has worked well with only minor problems over the 3 or so years I\u0026rsquo;ve had it running; I got spamhaused once, etc.\nThe problem all started with me trying to patch a perceived security hole. See, docker doesn\u0026rsquo;t respect firewalls like UFW or firewalld (all based on iptables of course), instead opting to allow ports through iptables as you add -p flags to your containers in spite of any other rules you may have.\nNow I thought this was rather terrible. I don\u0026rsquo;t want to have to look both at my firewall and at all my docker port bindings to check if something is open. So as many of us would do, I started trawling the internet for solutions and started to learn about why this behavior existed.\nAccording to some articles/stackoverflows/etc. the way to stop docker messing with iptables and creating its own rules is to disable the feature in the daemon.json. Seems simple enough. The only caveat that I found mentioned was that container networking would break (in terms of internet reachability) but that\u0026rsquo;s ok because I just had to add a firewalld rule to allow masquerading and that problem was solved.\nNow the problem I failed to see was that of NAT changing. Prior to disabling the iptables flag, the mailserver would see connections\u0026rsquo; IPs as their real public ones. However afterwards, every single IP was that of the internal docker network default route.\nI didn\u0026rsquo;t think much of it at the time, merely that it would be more annoying to see who was connecting but that was fine because I had what I wanted. Firewalld was now the sole controller of my ports 🎉\nLittle did I know (or maybe I did and just forgot) that postfix has a trusted list of IPs and it will relay anything from them without question. These IPs include internal IPs such as that of the default route\u0026hellip;\nSo essentially every SMTP request was being NATed to have a sender address of 172.22.1.1 and postfix started sending EVERYTHING 😵‍💫\nIt wasn\u0026rsquo;t long before a plethora of bots had saturated my poor NUC with HUNDREDS OF THOUSANDS of emails.\nI got home this evening to lag spikes in Tarkov which prompted me to check the server where I found this mess.\nAfter taking everything down, re-enabling the iptables and flushing all the postfix queues, I was able to spin back up and not have the whole thing start spiralling again.\nSome tips for those hosting mailservers:\nUse a mail server checker like https://mxtoolbox.com/SuperTool.aspx Setup monitoring and alerts for server CPU usage/spikes in requests/etc Don\u0026rsquo;t fix what ain\u0026rsquo;t broke I\u0026rsquo;m gonna go cry myself to sleep now and pray that the big mail hosts like Google and Microsoft take pity on me and my screw up. (We all know I\u0026rsquo;ll never be able to send another email to Microsoft again, who am I kidding)\n","date":"29 November 2023","externalUrl":null,"permalink":"/posts/email-fail/","section":"Posts","summary":"A mailserver incident post-mortem.","title":"Email Fail","type":"posts"},{"content":"","date":"29 November 2023","externalUrl":null,"permalink":"/tags/firewall/","section":"Tags","summary":"","title":"Firewall","type":"tags"},{"content":"","date":"29 November 2023","externalUrl":null,"permalink":"/tags/incident/","section":"Tags","summary":"","title":"Incident","type":"tags"},{"content":"","date":"29 November 2023","externalUrl":null,"permalink":"/tags/postfix/","section":"Tags","summary":"","title":"Postfix","type":"tags"},{"content":"","date":"29 November 2023","externalUrl":null,"permalink":"/tags/trust/","section":"Tags","summary":"","title":"Trust","type":"tags"},{"content":"","externalUrl":null,"permalink":"/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"},{"content":"","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"}]