MadaMada
@madamada@snac.void.my
Its never the final version :)
Its just I added there things I need from it - if something new/else arises - then I am free to add/update it.
What other features from jmore(8) would You like to see?
Its kinda a good idea.
I would leave rctl(8) settings to /etc/rctl.conf file - but just to DISPLAY used CPU and RAM is a good idea. I will try to make that possible in the following month.
Ping me if its not ready on 1st of September please :)
I only upgraded the IPv4 one.
I only use IPv4 and there is no place I use IPv6 - I included/added IPv6 version because someone may need it - and if someone is willing to update it based on my IPv4 update then sure - I will also upload the upgraded version.
Politics is theater. The current show is off-off-off-off-off-Broadway.
https://comam.es/snac-doc/snac.8.html#Migrating_from_snac_to_Mastodon
and this
https://comam.es/snac-doc/snac.8.html#Migrating_from_Mastodon_to_snac
If you want to move from one #snac to another, it's mostly the same.
If you want to change the domain, you must move all your accounts from the old server to the new one, while both instances are operative.
Add the following to named.conf or named.conf.options
// BIND9 configurationSave the configuration and reload named: rndc reconfig .. Now all requests to the upstream DNS servers will use TLS 🙂
//
// TLS upstream servers
tls cloudflare-tls {
remote-hostname "one.one.one.one";
protocols { TLSv1.3; };
};tls opendns-tls {
remote-hostname "dns.opendns.com";
protocols { TLSv1.3; };
};tls dns-sb-tls {
remote-hostname "dns.sb";
protocols { TLSv1.2; TLSv1.3; };
};options {
...
forwarders port 853 {
2620:119:35::35 tls "opendns-tls";
2620:119:53::53 tls "opendns-tls";
208.67.220.220 tls "opendns-tls;"
208.67.222.222 tls "opendns-tls";
2606:4700:4700::1001 tls "cloudflare-tls";
2606:4700:4700::1111 tls "cloudflare-tls";
1.1.1.1 tls "cloudflare-tls";
1.0.0.1 tls "cloudflare-tls";
2a09:: tls "dns-sb-tls";
2a11:: tls "dns-sb-tls";
185.222.222.222 tls "dns-sb-tls";
};
forward first;
};
Oh well, I've tried to install FreeBSD 14.3 on an old PC of mine, but after the first reboot, it would always get stuck on the POST screen and I also couldn't enter the BIOS, as long as the disk with FreeBSD on it was connected.
The installation from a USB drive itself went fine, but wouldn't boot no matter what. It's one of my first PCs with an AMD Athlon 64 3000+, an MSI K8T Neo2, 1 GiB DDR, a 80 GB Seagate ATA HDD and a ATI 9800. I suppose the Phoenix BIOS doesn't like FreeBSD's MBR?
@madamada I just did that, because it was booting fine from the USB drive. Even managed to install FreeBSD 15.0-CURRENT on this oldtimer (UFS). 😎
@madamada It boots off USB thumb drives and USB SSDs, but not from ATA or SATA drives connected to the motherboard.
With the USB SSD, even ZFS works with GPT (BIOS).
@madamada For that I need to put some jumpers on the HDD, right now I dont have any. 😅
For now it's more than enough to use an external USB SSD for the job.
Need another jail like that one? 
see: bastille clone help
This lets you duplicate containers in seconds. Fast, efficient, exact.
@madamada Yes! Thin, thick, clone, empty (experiments) and Linux (Debian and Ubuntu) on both UFS and ZFS.
Countries, where all mobile networks provide #IPv6 for data use.
de (Telekom, Vodafone, O2, 1&1)
fr (bouygues, free, orange, sfr)
cz (O2, Vodafone, T-Mobile)
partly:
at(a1, magenta) drei?
in (jio) ??
us (T-Mobile, Verizon, AT&T, ?) ?
Is there a overview?
Can you confirm that your mobile ISP in your country supports ipv6?
Besides the other adventures I also just installed FreeBSD on a new dedicated server which will likely become my new internet focal point.
I realized that there is too much complexity in the current setup to save it in of itself.
So I‘ll go with rebuilding it on a different machine, so I can set the speed.
FreeBSD because I finally want to use it in production and want native ZFS.
I’ll hopefully be able to run Jails for stuff that’s easy to run directly and create an Alpine Bhyve VM with Docker and a crapton of docker-compose files for the rest.
Simple, easy, effective (hopefully)
Now trying to figure out how to create a working networking stack for jails and VMs.
IPv6 is pretty simple, just move the prefix to a bridge and connect stuff to it.
But v4 is being the problem child again, because I'll need to do NAT shenanigans and I don't yet know how to make both work at the same time.
I'd now how to on Linux, but I'm still figuring out the FreeBSD ways. It's fun though :3
@jana Maybe you could g give the bridge a IPv4 address and set it as the default gateway for the jails?
@madamada @subnetspider Hmm, yeah, that‘s how I would normally imagine it too. Now just gotta figure out the chain of rc parameters to make that happen :D
Jail Networking turned out way easier than expected.
I just used the default `bastille0` interface that `bastille setup` created and assigned my container a private IPv4 & a public IPv6. And both just worked immediately.
That was so stupidly simple.
Now I just need to do a similar thing for bhyve VMs, which I will setup next.
Now installing Caddy (unprivileged with macport) into a jail.
This will hopefully end up becoming the central TLS termination point for the whole server and in the process allow me to play around with QUIC :3
I'll redo the networking again to use VNET.
This shared interface thing works, but it's getting a little messy now with my firewalling.
Well, now we get into subnetting and I don't want to cutting up a /64, so now gotta wait for Hetzner to give me my additional /56 on this machine.
Could I try to get this into my space right from the start? Sure.
But is that in the spirit of keeping it simple? No.
And as the /56 is free, except for a setup fee, why not take it anyways.
/56 assigned.
Jailing shenanigans can commence.
After a lot of trial and error I managed to setup a jail with working v4 and v6 connectivity.
Right now I still have to manually set defaultrouter= in order to make v4 connectivity work, but v6 is already handled automatically by SLAAC.
FreeBSD acts as the router, taking packets between bridge0 to em0.
I'd say this is looking promising so far and I think I managed to get a better understanding of the FreeBSD network stack, rc.conf, etc.
But enough of that for today.
Progress! I managed to set the defaultrouter via the bastille_network_gateway parameter in the bastille.conf.
The generated inet parameters for dual stack are still wrong. According to a GitHub issue that's already fixed, but because I'm installing via pkg, my version is too old.
I have decided to just manually fix it for now. I don't wanna mess with ports on the host.
Then I had to slightly change my portacl config, because in a VNET jail I also need to set net.inet.ip.portrange.reservedhigh=0, but for that to work securelevel needs to be 0.
So now I set that in the jail.conf and raise it to 2 via rc.conf.
I re-added the rdr rules to the host pf.conf and that's working. Now I just need to configure the jails pf.conf. :3
It works!
And it does so on both IP stacks!
(Using Fedi as a distributed load test, nice)
And it even does QUIC now (forgot UDP 443 in the firewall)
Thanks Caddy :3
Next step: Try vm-bhyve to setup a VM and figure out how I want to do that networking.
Still undecided whether I want to try putting jails and vms on the same bridge, or whether I want to create a separate one. I'm leaning towards the second option right now.
And when that works, then my POC is operational and I can start putting some real workloads on there :3
I have started my vm-bhyve adventures.
After first misunderstanding the `switch` functionality a bit, I wondered why my networking wasn't working.
Now I found a working solution, though I might tweak it again and go full custom bridge that I just import. Mainly for full flexibility and also my own naming.
Also had a quick issue with only pings working, but no TCP/UDP. Turned out I forgot to allow the traffic in `pf.conf`.
Other guides and resources often wanted me to do some other `sysctl` parameters to skip bridge traffic from filtering, but I'd rather go full `pf`.
I managed a FreeBSD install, no problem.
Alpine was a little harder. I got it installed via grub, but then couldn't get it to boot, no matter what.
I saw suggestions to just use UEFI instead, so I did that and it worked beautifully.
I'd rather unify on UEFI anyways, so I take it.
I feel like I already have an okay grasp on how vm-bhyve works, though I'll try some more test installs, also with different OSes, to really get a grip on it, before I setup my final productive VMs (Alpine + Docker, Windows Server 2022 and some Debian VMs I need to figure out how to migrate from Proxmox).
So far, so good :3
This Alpine VM in bhyve is so freaking fast.
alpine-virt
bhyve-nvme
uefi
This thing rips.
And IPv6 isn't working yet.
Yet unsure whether the fault is on the Alpine or FreeBSD side, but I'll ignore it for now and first try some other OS installs.
Also, question to the #bhyve people around.
Should I be using disk image files or zvols?
I understand so far the zvols can give me better performance, but I don't yet understand the downsides.
Doing my first test install of Windows Server in a bhyve VM. (Yes, I need Windows Server. Regretably)
Absolutely eventless. Done in 5 minutes. Wow.
Also love that bhyve is using VNC for graphics **and** halts installs until VNC is actually connected.
Oh, right, there is WS2025 by now. Forgot about that.
Well, this was so quick, let's give that a try also.
Doing an upgrade in the process would be nice.
Before that: Let's switch to using ZVOL instead of a disk image.
I just learned that vm-bhyve can also create those automatically with the right template.
And yup. With a simple disk0_dev="sparse_zvol", it was automatically created and WS2025 is now installing to it.
This means I can super easily snapshot and backup individual VM datasets.
I'll have to play around with zfs send/receive for that. Never done it, but it's about time.
Looking up how I could migrate some VMs from Proxmox to bhyve.
Seems as simple as creating a cloned disk image from Proxmox and using that in bhyve. And if I want to also use zvol there, `dd` the .img to the VMs zvol.
Could it really be that easy?
Hmm. The Windows VM now had it's second complete freeze up while doing updates.
I am willing to blame this on Windows (It's Windows, after all), still need to figure out what's going on there.
@jana Have you read this blog post by @stefano before?
https://it-notes.dragas.net/2024/11/15/migrating-windows-vms-from-bios-kvm-to-uefi-bhyve/
Maybe this'll help?
@subnetspider @stefano I have not! That makes things even easier! Also I didn’t consider the BIOS to UEFI part, that’s also relevant to my case, very convenient.
I have achieved IPv6 connectivity in bhyve VMs.
I just discarded the interface that vm-bhyve created by itself, created my own bridge1 (mirroring the config from the interface I have for jails) and imported it into vm-bhyve as a manual interface.
I think the part that actually did it was supplying the interface with a v6 interface in rc.conf, but I prefer doing it myself anyways, so I'm happy with this.
I would likely also just use one bridge for Jails and VMs, but I prefer to keep things nicely separated :3
And done a successful Debian install.
That one was a little trickier than Alpine, as the Debian installer didn't listen on Serial by default, when doing an UEFI install.
I now just used the Grub loader method in bhyve and copied the parameters from the debian template, which worked.
I'm getting real close to being able to migrate workloads.
Thought about yolo migrating my Mastodon instance over to the new server right now.
But I went the reasonable route and first reduced the DNS TTL, so that switching IPs will hopefully be reasonably quick.
How responsible of me 
Realized that Mastodon is one of the workloads I can pretty easily migrate to a FreeBSD jail.
So first gonna spin up a test instance with that, which shouldn’t be too hard with the help of this guide by @stefano
https://it-notes.dragas.net/2022/11/23/installing-mastodon-on-a-freebsd-jail/
Just learned about Bastille templates with the Bastillefile.
Feels kinda like a Dockerfile and it can be applied to any running jail.
I feel like this could be a nice little balance between my desire to make things declarative but also wanting to keep it simple.
I‘ll play around with it a bit.
I rejected the idea of looking into Bastille templates. They sound nice, but I didn't want them to become another thing holding me back from just getting stuff running in Jails at all.
So backtracked to installing manually.
I did all of the preparation to move over Mastodon. It's installed and just waiting for me to migrate over the database, change DNS and enable the new Caddy reverse proxy.
But don't have time for that anymore right now, so that will happen later :3
Well, gonna attempt to do the migration now.
See you on the other side fedi :3
Testing, testing.
Fedi, please respond.
Nice, it seems to be working just fine :3
My Caddy access log is going crazy from server trying to catch me up.
So this instance is now sitting in a nice and cozy FreeBSD jail :3
I'll be monitoring for issues, of course, but I'll carefully consider this a success.
Even managed to migrate my Redis rdb to Valkey, which I first intended to just ignore. Also worked fine.
Also did a PostgreSQL upgrade from 15 to 16 with it.
All the upgrades.
Next I'll probably look into re-enabling some sort of OpenSearch for it.
I disabled it, because it took too much RAM and didn't feel that useful anyways.
I've missed it so many times since that decision.
Terrifying part of it:
That's another machine that I put into production 
Enough computer touching for today 
The quest continues:
Currently learning how to write an rc script, because I want to try deploying pocket-id to use as a central auth source for further deployments.
pocket-id directly provides a FreeBSD binary, but no rc script or further FreeBSD instructions. All it needs is some environment variables though, so it should be easy enough to do.
Managed to create an rc script with daemon to start Pocket ID.
As Pocket ID is configured via environment variables, I am currently just doing that directly via _env rc variables.
Don't know if that is a good way to do it. Could probably map some of them to rc variables, if I wanted to?
Oh, rc.subr provides a ${name}_env_file option.
That's better.
Now that I finished my rc script, I looked if there is already an existing port I could use instead and yup, there is.
I explicitly didn't check beforehand, as I wanted to have the learning experience.
Comparing notes between my script and the one someone wrote in ports, I found a few differences where I don't know what approach makes more sense.
My script uses daemon and the ${name}_user variable to run the whole thing as an unprivileged user.
The script in ports uses a different variable and has daemon take care of changing the user via daemon -u '${pocket_id_runas}'.
https://cgit.freebsd.org/ports/tree/www/pocket-id/files/pocket-id.in
This runs daemon as root and pocket id as the user, while my setup also runs daemon as that user.
I use an install in the precmd to have the permissions work with the log directory.
Now I wonder which approach makes more sense.
Maybe @stefano, do you have any input on that?
Feature request #IPv6 (again)
First comment 🤦
https://meta.stackoverflow.com/questions/434456/stack-overflow-still-not-reachable-via-ipv6
@tschaefer Previously login and viewing question was not working when IPv6 was forced.
Now only the data dump page errors out.
Not sure they added support IPv6 or not, because https://data.stackexchange.com show a Private IPv4 address.
[root@serv ~]# ndp -na | grep fe80::1Grr...
[root@serv ~]# netstat -rn6 | grep default
default fe80::1%eth0 UGS eth0
[root@serv ~]# ping -6 -c2 -t2 one.one.one.one
PING6(56=40+8+8 bytes) 2a02:c207:xx:xx::1 --> 2606:4700:4700::1001--- one.one.one.one ping6 statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss[root@serv ~]# curl -v6I https://one.one.one.one/
* Host one.one.one.one:443 was resolved.
* IPv6: 2606:4700:4700::1111, 2606:4700:4700::1001
* IPv4: (none)
* Trying [2606:4700:4700::1111]:443...
^C
Update: After trial and error, a static ndp entry is needed to recognize their shitty gateway..
ndp -s fe80::1%eth0 gw:mac:addr
ff02::2%eth0 doesn't return any neighbour routers, ndp -na doesn't show any neighbours .. simply adding fe80::1%eth0 won't work right away, so we have to resort to tricks like this to get IPv6 going.. grr
@madamada If only I would have known about static ndp entries 2 years earlier ... back then I lost IPv6 connectivity on my VPS regulary every 4 weeks or so because the gateway (fe80::1%vmx0) disappeared from the neighbor discovery table...
CC: @TomAoki@bsd.cafe @stefano@bsd.cafe @NebulaTide@bsd.cafe