Originally Posted by imatlas
In a strictly literal sense it appears that you are right; factory predates manufactory by some time, but had a rather different meaning until at about the time when manufactory emerges by a compounding of and manufact (handmade) with -ory (place), to mean, depending on context, "something that is made by manual labor", "the production of manufactured goods, and "the place where things are made by hand". Contemporary with this meaning, "factory" evolves from "office for agents (factors), or "real estate office" to come to mean "a place where things are made". It would be rather strange if this change arose spontaneously without any connection to the concurrent evolution of a closely related word.
The use of 'factory' as a place for munufacture predates te use of 'manufactory' as a place of manufacture.
5. a. A building or range of buildings with plant for the manufacture of goods; a manufactory, workshop; "˜works'.1618
USSHER Let. to Camden 8 June, The Company of Stationers in London are now erecting a Factory for Books and a Press among us here.
2. a. A place where a product is manufactured; a factory or workshop, now esp. one where articles are handmade.1641
Acts Parl. Scotl. (1870) V. 657 For intertayneing and erecting of Manufactories for cloathe... Forsamelke as the want of manufactories.