Is there anything Grafana cant do?
I have so many things pumping data “into” Grafana these days I’m surprised they haven’t tried to force me to pay for an enterprise license.
Anyway, thanks for sharing these, @ruud@lemmy.world. As a performance engineer, I love to see this level of detail and commitment on your part to keep the user experience for lemmy.world at acceptable levels.
Easy one to miss! The documentation isn’t very detailed. 🙂
What is your domain set to?
Also I think the !selfhosted@lemmy.world community might be a better place to ask for help.
That’s most definitely a bug, in my opinion. Might want to file an issue on GitHub.
As a Performance Engineer myself, these are the kind of performance improvements I like to see. Those graphs look wonderful. Nice job to all.
Neat! I’ll definitely check this out.
Oh, this span builder error. Yeah I’ve been seeing that error since I upgraded to 0.18.0. Sorry, I don’t have a solution for this one.
Glad you figured out your pictrs error though! One other thing I did was update my resolve.conf so that valid DNS hosts were making it to the containers. Also try rebooting your host.
Feel free to add to this discussion. https://github.com/LemmyNet/lemmy/issues/3314
Yay! Thank you for confirming
Yay! Small victories
There was a pretty big jump in active instances, too!
Active daily user count is about 50k.
Are you running lemmy 0.18.0? It breaks a bunch of internal network connectivity. The short term solution is to add an external network to both the lemmy-ui and pictrs containers, then you can change http://pictrs:8080 to http://your_domain.
Note: there are also some issues in 0.18.0 with comment federation at the moment. Please reply to confirm this comment was seen, thanks!
Yes. It’s called performance testing. Basically an engineer would need to setup test user transactions to simulate live traffic and load test the system to see how everything scales, where it breaks, etc. Then you can use the results of the tests to figure out how big of an instance you should use for your projected number of users.
Jmeter, and locust.io are the two biggest open source performance test tools.
The alternative is take a wild guess. See how the system behaves, and make adjustments in real time… like what @ruud@lemmy.world is currently doing.
That was fast…
Ideally, yes. If that can be the reality, and I suppose that is how it should would with federation, then server costs should never get out of hand.
You bring up a very good point. Currently lemmy.ml has thousands of users. Lemmy.world has thousands of users. The hardware they have selected to run their instances is adequate for now, but, what is the plan for scaling out if the user base grows? Is there one? They have a donation page on each lemmy instance (click or tap the heart icon,) but that can’t be enough to pay for the cost of running something used by millions of people, even if only 100s of thousands are ever only online at any given time.
In terms of UI/UX, @dessalines@lemmy.ml has mentioned in a post they are currently working on major performance improvements and enhancements.
3.2k users is impressive. But yes, 10 seconds is bad. 2.5 seconds is about the max I find acceptable. If you don’t mind me asking, what are you running on, in terms of hardware?
I’m tempted to fire up jmeter, or locust, or k6 and push some load against my instance to see how things look internally under load.
Thanks for this post, @ruud@lemmy.world. I’ve setup my own instance and am enjoying the lemmy experience so far.
At least the latest RC cut the post and save time in half, for me. That’s a measurable improvement. It definitely must be due to server load, as posting and saving to !lemmyworld@lemmy.world from my instance seems pretty snappy.
Yes, there is: 0.18.2-rc.1, which has the hot fix, but will also require a DB query to “fix” the modlog once upgraded.