Why is LS1Tech so SLOW???
Its going to be tough for me to stress test vBulletin-PostgreSQL as well.. Its going to be tough to get 1000+ clients on the test bed at once to hammer it. Whats worse, is my dev server at the house isn't exactly the same performance level as our production servers 
An old AMD Athlon 1800+ with 1.5GB mismatched ram versus a dual 3.0GHz XEON with 4 gigs of ram. I'd LOVE to be able to build an Opteron box 
but my money is earmarked elsewhere.
An old AMD Athlon 1800+ with 1.5GB mismatched ram versus a dual 3.0GHz XEON with 4 gigs of ram. I'd LOVE to be able to build an Opteron box but my money is earmarked elsewhere.
Originally Posted by technical
Some, but not all. Best case example is Apache. Many Java technologies fall into that category as well e.g. Struts.
Actually two months is not plenty when i've got a year-long senior design project as one of the other 5 classes (which requires 20 hours of work a week) as well as grad computer security, networking, plus other undergrad req's.
BTW - any good tutorials on getting a struts app up and running? You've gotta use Ant to compile them into a .war file before deploying them in the webapps folder right? I'm using Apache Tomcat server.
BTW - any good tutorials on getting a struts app up and running? You've gotta use Ant to compile them into a .war file before deploying them in the webapps folder right? I'm using Apache Tomcat server.
Staging Lane
Joined: Nov 2001
Posts: 62
Likes: 0
Originally Posted by Brains
CF does a lot of archiving, which moves things out of the main post and thread tables. Its kind of a pain in the butt to search because of it, and is the main reason we've been hesitant to follow suit. With me converting all the queries to be SQL92 compliant, it opens us up to run any database server that PHP offers support for (in one form or another, be it a library module or ODBC). There's no abstraction layer in the code however, so the conversion is MUCH more involved than simply changing MySQL-specific language/functions in the queries. I'm having to alter a *LOT* of the source because they coded directly against MySQL and its unique way of doing things. For instance, getting the last inserted auto_increment'ed column's value for a record -- MySQL has a function last_insert_id(), that will pull the last ID without having to specify which table you want the value from. Nobody else is that "lazy" about it, so the code should be changed to specify -- and there's a LOT of locations to do so.
Then there's performance tuning differences between database engines. One of the reasons why switching over to even InnoDB on MySQL doesn't pan out well, is the use of the ever-so-common SELECT COUNT(*). With the MyISAM storage engine, this isn't a costly call at all -- it executes instantaneously because with table locking the number of records is always the total number of rows in the table. With a storage engine that supports multi-level concurrency, you could have any number of records in flux -- inserts, deletes, etc. So, a COUNT(*) is very costly - especially on large tables. vBulletin makes heavy use of COUNT(*) as well, almost every page in fact. That will have to be rewritten as well.
The only "easy" solution I've come up with, and its one I may very well do first, is to segment the site. Basically turn the forums into a group of forums. vBulletin stores all the posts in one table, and all the threads in one table. That means we have over 3 million rows in the post table. It also means the whole thing is locked on updates, which is why the entire site freezes up when locks are held. By segmenting, I can reduce that to one pair of post/thread tables per group of forums, or possibly per forum. That would reduce the number of rows considerably, and severe lock times would only affect one or a small set of forums.
Then there's performance tuning differences between database engines. One of the reasons why switching over to even InnoDB on MySQL doesn't pan out well, is the use of the ever-so-common SELECT COUNT(*). With the MyISAM storage engine, this isn't a costly call at all -- it executes instantaneously because with table locking the number of records is always the total number of rows in the table. With a storage engine that supports multi-level concurrency, you could have any number of records in flux -- inserts, deletes, etc. So, a COUNT(*) is very costly - especially on large tables. vBulletin makes heavy use of COUNT(*) as well, almost every page in fact. That will have to be rewritten as well.
The only "easy" solution I've come up with, and its one I may very well do first, is to segment the site. Basically turn the forums into a group of forums. vBulletin stores all the posts in one table, and all the threads in one table. That means we have over 3 million rows in the post table. It also means the whole thing is locked on updates, which is why the entire site freezes up when locks are held. By segmenting, I can reduce that to one pair of post/thread tables per group of forums, or possibly per forum. That would reduce the number of rows considerably, and severe lock times would only affect one or a small set of forums.
I have a resource intensive query that will tally up the most commonly indexed words in your index table which will allow you to prevent some useless words from being indexed by putting them in badwrods.php and to go ahead and remove them from the search index which will have to be compacted afterwards.
The thread previews and dots are another resource hog. I rewrote the thread previews on ls2.com to elimintate yet another query into the post table by storing the preview in a newly added field in the thread table. And you can add a very simple logic to temporarily turn off the dots when the loads go up.
I would be a fool to assume you are not using a php accelerator and query caching in MySQL.
These ideas for as little as they may sound, will make a noticeable difference.
I'm actually not using the postindex table at all -- zero rows I've switched over to using FULLTEXT indexes on the post and thread tables which helps immensely.
I am continually trimming the MySQL caches and buffers, there's not much else there I believe. I'm running a PHP cache/accelerator, there's no way this site would even run without one
I've switched over to using FULLTEXT indexes on the post and thread tables which helps immensely. I am continually trimming the MySQL caches and buffers, there's not much else there I believe. I'm running a PHP cache/accelerator, there's no way this site would even run without one
Staging Lane
Joined: Nov 2001
Posts: 62
Likes: 0
Originally Posted by Brains
I'm actually not using the postindex table at all -- zero rows I've switched over to using FULLTEXT indexes on the post and thread tables which helps immensely.
I am continually trimming the MySQL caches and buffers, there's not much else there I believe. I'm running a PHP cache/accelerator, there's no way this site would even run without one
I've switched over to using FULLTEXT indexes on the post and thread tables which helps immensely. I am continually trimming the MySQL caches and buffers, there's not much else there I believe. I'm running a PHP cache/accelerator, there's no way this site would even run without one
You're in a pretty tough situation.
Take a look at how thread previews are done. They are not that difficult to rewrite.
Let me know if there is anything I can do to help. I'll PM you my number if you need to get a hold of me.
LS1 Tech Stories
The Best V8 Stories One Small Block at Time
Gas Monkey Built a 6-Wheel Ferrari Testarossa With a Corvette LT4 Engine
Verdad Gallardo
7 Most Reliable High-Performance Engines GM Has Ever Built
Verdad Gallardo
Amazing '71 Camaro Restomod Is Modern Muscle Car Under the Skin
Verdad Gallardo
6 Common C5 Corvette Failures and What's Involved In Repairing Them
Pouria Savadkouei
Retro Modern Bandit Pontiac Trans AM Comes With Burt Reynolds' Autograph
Verdad Gallardo
Top 10 Greatest Cadillac V Series Performance Models Ever, Ranked
Pouria Savadkouei
Top 10 Most Powerful Chevy Trucks Ever Made!
Hennessey's New Supercharged Silverado ZR2 Has 700 HP
Verdad Gallardo
Coachbuilt N2A Anteros Is an LS2-Powered C6 Corvette In Italian Clothes
Verdad Gallardo
TECH Addict
Joined: Mar 2004
Posts: 2,336
Likes: 0
From: Fat Chance Hotel
Originally Posted by ArcticZ28
BTW - any good tutorials on getting a struts app up and running? You've gotta use Ant to compile them into a .war file before deploying them in the webapps folder right? I'm using Apache Tomcat server.
I actually got it all figured out... just getting an error now which I can't for the life of me trace back. It's something about instantiating one of the TLD files or something:
'org.apache.jasper.JasperException: Unable to initialize TldLocationsCache: InputStream cannot be null...'
..and the list goes on. At any rate, if anyone has any input/insight on this please PM me. Other than that, I'll stop hijacking this thread
'org.apache.jasper.JasperException: Unable to initialize TldLocationsCache: InputStream cannot be null...'
..and the list goes on. At any rate, if anyone has any input/insight on this please PM me. Other than that, I'll stop hijacking this thread
I'm just throwing this out here, not sure if this has ever been thought of but.....
Why don't we split the website up and do the saving/writing on different servers. So, like moving all the LSX stuff to one server....LTX stuff to another, Apperance/Maintenance, ect....you get the idea....and then have one main page that doesn't have to be written to and locked up whenever one person makes a post, which links to the page that contains the corresponding data, which is on another server. If I were to draw a diagram of this, there would be the Main page, just like the current mainpage....but doesn't give all the updated stats, its just a main page framework with all the links, basically looks exactly the same it does now, maybe its updated every 10 mins or so. Then, each sub-catagory is contained on seperate servers, which is linked from the main page. So when you click on lets say....External Engine in the LS1/LS6 section....it takes you to the same listing, except now you're viewing a page that is independent of all the rest of the main categories of the site, and only a post in the LS1/LS6 section will have to be written to that particular server, not hanging up the entire webpage and all other catagories which are un-related to that particular heading. Does this make sense, or is my rambling completely incoherent?
Regards,
Adrian
Why don't we split the website up and do the saving/writing on different servers. So, like moving all the LSX stuff to one server....LTX stuff to another, Apperance/Maintenance, ect....you get the idea....and then have one main page that doesn't have to be written to and locked up whenever one person makes a post, which links to the page that contains the corresponding data, which is on another server. If I were to draw a diagram of this, there would be the Main page, just like the current mainpage....but doesn't give all the updated stats, its just a main page framework with all the links, basically looks exactly the same it does now, maybe its updated every 10 mins or so. Then, each sub-catagory is contained on seperate servers, which is linked from the main page. So when you click on lets say....External Engine in the LS1/LS6 section....it takes you to the same listing, except now you're viewing a page that is independent of all the rest of the main categories of the site, and only a post in the LS1/LS6 section will have to be written to that particular server, not hanging up the entire webpage and all other catagories which are un-related to that particular heading. Does this make sense, or is my rambling completely incoherent?
Regards,
Adrian
I've thought long and hard about doing it that way, or by simply storing each forum in its own pair of thread/post tables, but there's a LOT of code that would also need to be written to keep the site functioning as "one." Instead of put the effort into "bandaids" for MySQL, I decided to put my time into re-coding vBulletin to use a better database backend completely. I currently have a fairly complete implementation of vB 3.5.0 running on a PostgreSQL backend, just going through and cleaning up some final bugs
Outstanding, I'm not sure how all of that works, it was just an idea that popped in my head when I couldn't get on the site for 5 zillion hours...haha. How does this new backend solve some of the performance problems that are being seen atm?
Regards,
Adrian
Regards,
Adrian
Originally Posted by Brains
I've thought long and hard about doing it that way, or by simply storing each forum in its own pair of thread/post tables, but there's a LOT of code that would also need to be written to keep the site functioning as "one." Instead of put the effort into "bandaids" for MySQL, I decided to put my time into re-coding vBulletin to use a better database backend completely. I currently have a fairly complete implementation of vB 3.5.0 running on a PostgreSQL backend, just going through and cleaning up some final bugs
The problem with MySQL (specifically the MyISAM data storage engine) is the lack of concurrency control. Simply put, if you make a change to a table, *EVERYTHING* else waits until the change is complete. Well, on a message forum with 3+ million posts online and over 1000 people making changes, you'll tend to have a few of them stack up -- so everything crawls to a halt while the DB tries to catch up. I looked at switching to the InnoDB engine, which DOES support concurrency, but other large forums have tried it with limited success.
So, that's why I looked to a better DB engine entirely, and the best open source package available is PostgreSQL.
So, that's why I looked to a better DB engine entirely, and the best open source package available is PostgreSQL.
im not gonna pretend i know anything about this stuff guysand hopefully i dont get flamed, but is it possible to split the forums onto two servers??? and if it came down to paying for a membership to speed this place up, you can count me in. as long as i get a free decal =)
phantom, that wouldn't help all that much.
the solution that is in the works will eliminate much of the problem, as Brains explained between the 3 or 4 posts of his which were right above yours.
the solution that is in the works will eliminate much of the problem, as Brains explained between the 3 or 4 posts of his which were right above yours.
Brian I'm only seeing < 400 people on tech and I think it's running slower than it's ever run before. Is it just me or are you doing something, or is something wrong? It's taking over 1 minute on dsl just to switch pages.
:EDIT: must be my internet because all of the sites I'm going to are taking forever. Sorry about that.
:EDIT: must be my internet because all of the sites I'm going to are taking forever. Sorry about that.