Unfiltered updates straight from the Ozot crew — what we're working on, what's coming next, and what's happening behind the scenes. No marketing polish, just the real stuff.
OZOT
Working on our optimal routing solution that makes sure IPs/prefixes are routing over the optimal path to be more scalable and work with the new network design better. I expect this one to be hard and a lot of playing around in the lab for me.
OZOT
Ended up letting the clanker write basically all of the code and then realised I understood basically none of it and I am now starting from scratch again where majority of the code will be mine, but I will obviously ask the clanker for help here and there. It will be something I understand in the end rather than what it was going to be.
OZOT
New hardware installed in Sydney in prep for the new network design implementation. I've shutdown the lowest latency path for the time being so I can implement the new network design without disruption to customers. Just waiting on a vendor to finalise some config on their end then I should be good to implement it. I'm hoping to have the new network design implemented for Perth <-> Sydney traffic in the coming days. This should drastically improve packet loss to Sydney.
OZOT
Ok third party vendor was very quick to update config required. Shoutout to EdgeIX the best IX in Australia of course ;)
OZOT
Now over to me to implement the new network design. I already got stuck on a little problem and on my afternoon walk clarity struck me and now I'm moving again. Getting closer and might have something im place by tonight, we shall see. If I get stuck again I will go to the gym to have clarity strike me again :3 uwu
Still going, now onto Epica - Aspiral (FULL ALBUM STREAM) https://youtu.be/BsKwwgfWX3Q?si=yCvVmgo0PfB1K1CX . Naming convention and numbering of this new network design is my main headache atm, I don't fully like the naming/numbering I have so far.
OZOT
Ok given I am slightly stunlocked on the name/numbering at this second I will go to the gym and hopefully gain some clarity as I lift. Will probs switch to Shadow of Intent while I am in there https://youtu.be/VmH3utJxoNw?si=b1syrTNI4U71hiVH and then back to Epica when I am back :)
OZOT
Ok I'm back and only trained one macro muscle group (chest) and no minors today because this was weighing on my mind. Good news, I think I have clarity on the name/numbering and can trinket the stunlock.
Ok, new network design Perth <-> Sydney mostly configured and looking good. We just give it a bit of time to confirm stability and do a few tests before bringing into PROD. If looking good should be in PROD tomorrow or the next day.
OZOT
Looking stable so far. Configuring the Singapore portion of it all now.
Onto some OG Nightwish, this is my least favourite Nightwish album, but the Youtube algo gave it to me so I listen https://youtu.be/mkPDHi9X_x0?si=ntNz_HyZ6D1XEqtc . Very close to being 100% happy with config and tests before bringing new network design into production.
OZOT
Ok all of the technical tests passed and stability is looking good. Should be good to bring the new network design into production. It is prime time right now so certainly not doing it now and will most likely set an alarm for 3AM and do it then. In thoery there should be no packet loss while enabling it, but will side on caution.
OZOT
Terrestrial/Subsea cable flapping stabilizer put in place for Perth <-> Sydney path.
OZOT
Ok new network design has been enabled and no packet loss was observed. Our terrestrial/subsea cable failover times should be a lot better on our Perth <-> Sydney path and I'll be monitoring closely over the coming days/weeks. I suspect no other ISP has gone through these lengths to get packet loss down to what I have on terrestrial/subsea link failure. Lots of secret stuff going on that I spent months figuring out in the lab. I was thinking about CPU tick rates and kernel level stuff for a long time...
OZOT
Ok, bad news, not looking 100% clean while in PROD. Getting a few 100ms packet drops every few mins. I have a theory on what is causing it and am getting more data to confirm my theory. I do have a plan to fix it if my theory is correct. I don't think the 100ms packet losses every few mins will be that noticeable so I'll leave it like it is for now.
OZOT
The universe loves to humble ya. Maybe I was getting to cocky/egotistical, plz forgive me universe :3 uwu I will get to the bottom of this
OZOT
Ok now gathering the data I need to be able to prove or disprove my theory. I will go to the gym while I wait for it to gather.
OZOT
I've taken the new network Perth <-> Sydney path out of play for the time being while I troubleshoot. Going variable by variable to find the root cause. My original theory has basically been ruled out (thank god because that would have been a Data Centre visit to fix) and the root cause is pointing to third party at this second, need to gather more data to be able to prove that and then reach out to them to hopefully fix.
OZOT
Interestingly the packet loss went away over the new network as soon as I moved traffic off of it (now it only has my test traffic over it). Jamming out to some Muse https://youtu.be/1J2_ojmncMA?si=lN4AVulnQCjVPKn3 while I investigate.
OZOT
Ok, I think my original theory might have been correct. I found a SFP module on the new network design path that has a suspicious amount of errors. I'll have to go to the Data Centre and physically change the SFP module and cable and see if that does the trick. Probs won't get around to this until tomorrow or the day after tomorrow.
OZOT
Ok, I replaced the SFP module with errors on it Tuesday night, I then re-enabled the new network design on Wednesday night, and it still didn't look as clean as I liked so I've disabled the new network again. It seems to correlete with traffic and I highly suspect third party at the moment. I've got some more experimentation to be do.
OZOT
This has highlighted a blindspot in my lab, I should have iPerf traffic generators in their as well, so if this is traffic related it would have been picked up in their too. I think there are ways around the problem, just requires more testing.
OZOT
Alright, it turns out I made a foreshadowing a few days ago and I've been humbled. It is looking like it was not an issues with a third party, but rather I've not optimized a setting correctly. I can't give away all of the secrets, but it is a area I am not what I consider a deep expert in and I think changing a few settings has done the trick. The new network has been re-enabled after doing throughput testing and it looks like the small amounts of packet loss have disappeared even under load. I'll be monitoring very closely, but yeah hopefully that is the new network design implemented for the Perth <-> Sydney leg!!! ggwp
OZOT
Certainly has been looking good now after these changes. Just made one more setting change for even more improvements, you gotta squeeze every penny https://youtu.be/l06Gh0RJK6Q?si=BN0L3cmsmxAlLyYk
OZOT
Researching a fog of war area of mine has made me do one more setting improvement to squeeze every penny. I think that is all I can do for now on this and it is looking pretty good aye.
OZOT
Looking good, now just gotta wait for a terrestrial/subsea cable failure so I can see the real world failover times on new network design.
OZOT
Working on a like-for-like latency and packet loss monitoring comparison between Aussie Broadband, Ozot, Telstra, and Superloop. I have a Proxmox server connected to a Switch that will then have an ethernet cable from the Switch to the nbn FTTP UNI-D1 (Aussie Broadband), UNI-D2 (Ozot), UNI-D3 (Telstra), and UNI-D4 (Superloop). Each VM on the Proxmox server will have the same CPU, RAM, etc as one another. They will all use the same physical uplink as one another (VLANs for the different UNI-D port). We will eventually make this latency and packet loss data publicly available. It will be monitoring latency and packet loss on a 100ms interval using UDP. It will be running to endpoints in countries around the world.
Certain Opticomm estates have limited available speed plans to the customer and it looks like we have a sign up flow bug where it will allow the customer to select a plan that is greater than the speed Opticomm can provide for their house/estate. Fixing this bug at the moment.
OZOT
Ok Codex has done its thing and this Opticom sign up flow bug should be fixed.
Singapore has fallen, don't game on Singapore game servers is my recommendation until at least one of these subsea cables is back online. I suspect the other subsea cables landing into Singapore are having a harder time. Packet loss to Singapore not looking great.
OZOT
Had a bug with our Speed Change via our Portal. It was using the wrong Plan IDs on the CURL POST and causing then to be rejected. Re-did the way we do the CURL POST for a more scalable solution. Should be better now.
OZOT
Understand this: things are now in motion that cannot be undone
OZOT
Making some good progress on the implementation of the new network design, should be implemented in the coming weeks for the Perth <-> Sydney path.
Jesus, who knew one of the worst tech experiences I would have in a long time is just trying to update the iDRAC firmware of a Dell PowerEdge in my lab... That wasted most of my day...
OZOT
Doing stuff in prep for new network design implementation. This stuff takes a while to progress when you need to ship stuff to other states and coordinate with third parties. Fun.
OZOT
Fk the cable Perth <-> Sydney has been dodgy lately. It looks like it had some issues tonight. Can't wait to implement new network design so I don't have to worry about cables be acting dodgy. Sooooooooooooooooon
OZOT
Ok I just made a network change to hopefully improve this until we wait until I am able to implement the new network design.
OZOT
On a different note, I was doing some Server shopping last night. The Server market is pretty interesting these days, Dell PowerEdge, HPE, Supermicro, ASRock Rack, GIGABYTE, ASUS and more.
OZOT
That DDR5 ECC RAM pricing though at the moment, bloody almost as much as the CPU.
OZOT
With the new network design I have implemented a stabilizer in the lab too. If the terrestrial or subsea cable is flapping (coming online/offline over and over) the stabilizer can only bring the cable back into production after x amount of minutes. In the real world we will do different minutes or even hours of stability depending on the terrestrial or subsea cable. Some certainly are more prone to flapping than others.
OZOT
My mind is obsessed with what I need to do to rollout my new network design when it probably should be more focused on how to market better. Engineers dilemma I guess, but better packet loss should improve organic marketing too so win-win I tell myself?...
OZOT
Alright, I just spent the past few hours trying to make improvements to my new network design that has better packet loss on terrestrial and subsea cable failures and my improvements didn't work. Infact they made the failover much worse. I was trying to change the new network design to something that works better in my mind at scale and those changes seems to have removed the packet loss improvements I found. I guess I have to keep the new network design as it is. It is fine, just wanted something that made more sense to my mind at scale and this new design is just something I will understand even better over time. Was worth a shot before fully commiting and rolling out the new network design in the coming months.
OZOT
Was listening to the new Muse album - The WOW Signal while playing around in the lab, rolling back the config now.
OZOT
Ok I think there was a bug with our Recruit a Friend system where the referrer was not getting credit, but the referred was. This should be fixed now. We shall see.
Not only have I found a way to improve packet loss on terrestrial or subsea cable failure in the lab, but I have also made scalability improvements to our proprietary routing solution that routes specific IP/prefixes via Countries/States. You can have all of the lowest latency terrestrial and subsea cables apart of your network, but if you are not using them for optimal routing then it is kinda pointless. I'll implement the proprietary routing solution after we have implemented the new network design in the coming months.
OZOT
Just found out information on a terrestrial cable that is showing potential. Will see how it pans out, secret stuff for now I guess.
OZOT
Do you think NBN will enter a death spiral with CPI increases each year? I'll be very curious to see the active services stats and if it is increasing or decreasing each year (even with new house builds and population growth). With 5G alternatives and Starlink alternatives my theory is at some point NBN will get into a death spiral where each CPI increase causes a percentage of the population to decide to move to alternatives or just use/hotspot their phones. This decrease in active services will cause NBN to have to increase the NBN service costs further pouring gas on death spiral? Will be interesting to see how it plays out over time. My theory is your average normie should be fine with 4G/5G on their phones and hot spotting it to devices they need to in time. How does Ozot fit into this theory though? Latency, jitter and packet loss will be best for gaming on fixed line so we should be ok as long as gamers choose us and we continue to provide the best latency, jitter and packet loss. On a side note, this System of a down song reminds me of a Ant Death Spiral for some reason https://youtu.be/GkPna4xPJmE?si=ME7EVIqurAenACR-
Not death spiral territory yet, but it certainly is a Zero Sum market and Peter Thiel wouldn't be proud https://youtu.be/3Fx5Q8xGU8k?si=ctJvZnHy1OIfOTX0 . In a zero sum market all you can really do is take customers from other providers to grow. You see providers doing this by having pricing wars getting to very very low margins and scale. All I care about is providing a network with the lowest latency, jitter, and packet loss though and I'm obsessed with that. Proving that brings us enough customers to be profitable as long as we are sort of competitively priced. Maybe we will have a NVIDIA moment in 10 years time after just focussing on providing the lowest latency, jitter, and packet loss globally some AI system will need that and boom we are the most important network. A bit of delulu can be the solulu.
OZOT
I've come up with a new network design that will improve our packet loss even more. I've been testing it in the lab and it has drastically reduced packet loss on terrestrial and subsea link failure. It might take a few months to actually implement this new network design in PROD, but it is certainly one of my higher priorities right now. Trying to get a few things sorted with third parties to make it a reality. Will be great to improve packet loss even more!
OZOT
One of the things that has triggered me to come up with a new network design that improves packet loss is due to the instability of the lowest latency cable Perth <-> Sydney. It was having issues about a week ago now and has always not been the most stable terrestrial link and I've been labing theories I've had on how to improve packet loss even more and I finally found something that drastically improved packet loss on terrestrial and subsea cable failures that certainly happen here and there. Fun times and will be happy once this new design is implemented in PROD, will post progress here I guess.
OZOT
To measure the packet loss improvements I had to build a secret tool using Rust, that was a project in itself and put me in a position to be able to measure the packet loss improvements down to the ms failover times. The other tools of the market for ms packet loss measurements aren't there right now...
OZOT
Bump, will replying to this make this thread at the top again? edit test 123
OZOT
Not sure if I jinxed it or not, but it looks like the lowest latency cable Perth <-> Sydney just had a few mins of not looking so great. Certainly highlights to me the need to get this new network design in place to reduce the packet loss drastically on terrestrial cable issues.
OZOT
untitled.mp3
OZOT
Got Claude Fable to redo our Website, Sign Up, and Portal, and Logo visuals. Hopefully our stuff doesn't look as bad anymore. We shall see what people think in time.
OZOT
Just made posts.php with the help of Claude Fable. The intention of this is to be like a Facebook, Instagram, X feed for basically shitposting. They might be long, they might be short, they might have text and images, they might have text and videos, they might be mp3 audio, not sure how this will be used at the moment. Probably network updates and progress tbh. We shall see.
